Oct 30 18:37:47.912816 ip-10-0-131-68 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:37:48.266552 ip-10-0-131-68 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:37:48.266552 ip-10-0-131-68 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:37:48.266552 ip-10-0-131-68 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:37:48.266552 ip-10-0-131-68 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:37:48.270669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270590    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:48.270669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270668    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:48.270669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270672    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270676    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270679    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270682    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270685    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270687    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270690    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270693    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270696    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270698    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270701    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270703    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270710    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270714    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270716    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270719    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270722    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270724    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270728    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270730    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:48.270762 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270733    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270735    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270738    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270741    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270743    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270746    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270749    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270752    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270754    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270757    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270761    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270764    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270766    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270770    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270772    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270775    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270778    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270781    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270783    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:48.271250 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270786    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270789    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270791    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270794    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270796    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270799    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270801    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270804    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270806    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270809    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270812    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270814    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270817    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270819    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270822    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270824    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270827    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270829    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270833    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270835    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:48.271731 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270838    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270841    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270843    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270846    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270849    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270852    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270855    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270857    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270860    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270862    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270865    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270868    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270871    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270873    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270876    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270878    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270881    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270883    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270885    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:48.272195 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270888    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270892    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270894    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270897    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270899    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270901    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270904    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270906    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270909    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270912    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270914    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270918    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270920    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270923    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270926    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270930    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270934    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270937    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270940    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:48.272692 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270943    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270945    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270950    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270954    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270956    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270959    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.270962    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271728    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271735    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271738    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271742    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271744    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271747    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271750    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271752    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271755    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271757    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271760    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271762    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271765    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:48.273124 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271767    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271770    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271772    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271775    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271779    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271782    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271784    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271787    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271790    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271793    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271796    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271798    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271801    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271803    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271806    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271808    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271811    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271813    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271816    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:48.273657 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271818    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271821    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271824    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271826    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271828    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271831    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271833    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271836    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271839    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271841    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271844    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271846    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271849    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271851    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271854    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271856    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271858    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271862    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271865    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271867    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:48.274192 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271870    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271872    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271875    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271877    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271880    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271882    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271885    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271887    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271890    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271892    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271895    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271897    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271900    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271903    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271906    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271908    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271911    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271913    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271916    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271918    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:48.274711 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271920    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271923    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271926    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271929    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271931    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271934    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271936    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271939    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271941    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271946    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271951    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271954    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271957    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271960    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271963    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271965    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271968    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271970    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271973    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:48.275191 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271976    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271978    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271981    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271983    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271986    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271988    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271990    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271993    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.271996    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272001    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272004    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272007    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272009    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272012    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272014    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272546    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272553    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272557    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272561    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:48.275669 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272563    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272567    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272569    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272573    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272576    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272579    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272582    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272585    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272588    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272590    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272593    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272596    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272599    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272601    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272604    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272608    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272611    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272614    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272617    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272620    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:48.276102 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272623    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272625    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272628    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272631    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272633    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272636    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272639    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272643    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272647    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272650    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272653    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272655    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272658    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272661    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272663    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272666    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272669    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272672    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272674    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272677    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:48.276594 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272679    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272682    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272684    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272687    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272689    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272692    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272694    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272697    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272700    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272702    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272705    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272707    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272711    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272714    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272717    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272719    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272722    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272724    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272727    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:48.277056 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272730    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272733    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272735    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272738    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272740    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272742    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272745    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272747    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272754    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272756    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272759    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272761    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272764    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272766    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272769    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272771    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272774    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272776    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272779    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:48.277517 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272782    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272784    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272787    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272789    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272792    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272794    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272797    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272800    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272802    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272805    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272808    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272810    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272813    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272816    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272818    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272821    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272823    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272825    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272828    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:48.277959 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272831    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272834    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272838    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272841    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.272843    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273434    2570 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273447    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273453    2570 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273459    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273464    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273467    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273471    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273476    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273479    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273482    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273485    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273502    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273506    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273509    2570 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273512    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273516    2570 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273519    2570 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273522    2570 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273528    2570 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:37:48.278396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273531    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273534    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273538    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273541    2570 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273546    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273549    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273552    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273555    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273559    2570 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273562    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273567    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273570    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273573    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273578    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273581    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273583    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273586    2570 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273590    2570 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273593    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273597    2570 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273600    2570 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273603    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273607    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273610    2570 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:37:48.278962 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273614    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273617    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273621    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273624    2570 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273627    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273630    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273633    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273636    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273640    2570 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273643    2570 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273646    2570 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273650    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273653    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273657    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273660    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273663    2570 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273666    2570 flags.go:64] FLAG: --help="false"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273670    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273674    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273677    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273681    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273684    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273687    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:37:48.279549 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273690    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273693    2570 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273696    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273699    2570 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273702    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273705    2570 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273708    2570 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273711    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273713    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273718    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273721    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273724    2570 flags.go:64] FLAG: --lock-file=""
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273727    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273730    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273733    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273738    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273741    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273745    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273748    2570 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273751    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273755    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273757    2570 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273760    2570 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273765    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273768    2570 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:37:48.280110 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273772    2570 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273775    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273779    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273782    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273785    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273788    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273791    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273794    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273801    2570 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273804    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273807    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273810    2570 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273813    2570 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273816    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273820    2570 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273822    2570 flags.go:64] FLAG: --port="10250"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273827    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273830    2570 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02a1af29f821041ef"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273833    2570 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273836    2570 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273839    2570 flags.go:64] FLAG: --register-node="true"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273842    2570 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273846    2570 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273849    2570 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273852    2570 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273856    2570 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:37:48.280743 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273861    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273864    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273867    2570 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273870    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273873    2570 flags.go:64] FLAG: --runonce="false"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273876    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273879    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273882    2570 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273887    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273890    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273893    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273896    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273899    2570 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273902    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273905    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273908    2570 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273912    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273915    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273918    2570 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273921    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273926    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273929    2570 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273933    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273937    2570 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273940    2570 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:37:48.281342 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273943    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273946    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273949    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273952    2570 flags.go:64] FLAG: --v="2"
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273957    2570 flags.go:64] FLAG: --version="false"
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273961    2570 flags.go:64] FLAG: --vmodule=""
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273965    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.273969    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274123    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274127    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274130    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274134    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274136    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274139    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274142    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274146    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274149    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274151    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274154    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274156    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274159    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:48.282012 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274161    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274164    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274167    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274169    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274172    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274174    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274177    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274180    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274183    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274186    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274189    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274191    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274194    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274196    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274199    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274202    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274205    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274207    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274210    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274216    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:48.282838 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274218    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274221    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274223    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274226    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274228    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274231    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274234    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274236    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274250    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274253    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274255    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274258    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274261    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274263    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274266    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274268    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274271    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274273    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274276    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274278    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:48.283354 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274283    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274285    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274288    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274290    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274293    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274296    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274300    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274304    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274306    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274309    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274311    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274314    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274318    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274322    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274325    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274328    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274331    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274333    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274336    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:48.283936 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274338    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274341    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274343    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274346    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274349    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274351    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274354    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274356    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274359    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274361    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274364    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274366    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274369    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274372    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274375    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274377    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274380    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274383    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274386    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274388    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:48.284392 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274391    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274393    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274396    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274398    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274401    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274403    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274406    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274408    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274411    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274414    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274416    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274419    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274422    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274424    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.274451    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274503    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:48.284877 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274511    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274516    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274520    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274523    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274526    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274529    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274531    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274534    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274536    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274540    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274543    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274546    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274549    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274551    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274554    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274556    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274559    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274561    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274564    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274566    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:48.285261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274569    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274571    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274574    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274577    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274580    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274583    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274585    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274588    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274590    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274593    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274596    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274598    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274601    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274603    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274605    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274608    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274610    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274613    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274616    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274618    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:48.285821 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274620    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274624    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274627    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274630    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274632    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274635    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274637    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274640    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274642    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274645    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274647    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274650    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274652    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274655    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274658    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274660    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274663    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274665    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274668    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274672    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:48.286283 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274675    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274678    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274680    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274683    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274685    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274688    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274690    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274693    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274695    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274698    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274700    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274703    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274705    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274709    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274712    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274714    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274717    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274720    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274722    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274725    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:48.286772 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274728    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274731    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274733    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274736    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274738    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274741    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274744    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274746    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274749    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274752    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274754    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274758    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274761    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274763    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274766    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274768    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274771    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274774    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274777    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:48.287235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274781    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274784    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274787    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274790    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274792    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.274797    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.281189    2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.281203    2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.281218    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.281222    2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281283    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281289    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281292    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281295    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281298    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281301    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281303    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281306    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281309    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281312    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:48.287739 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281314    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281317    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281320    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281323    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281326    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281329    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281332    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281334    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281337    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281339    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281342    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281345    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281348    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281350    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281354    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281357    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281360    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281364    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281368    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:48.288336 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281371    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281374    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281376    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281379    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281382    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281385    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281387    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281390    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281393    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281396    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281398    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281401    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281403    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281406    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281409    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281411    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281414    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281417    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281419    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281422    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:48.288818 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281424    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281427    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281429    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281432    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281434    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281437    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281440    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281443    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281445    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281448    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281451    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281454    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281457    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281459    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281462    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281464    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281467    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281470    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281472    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281475    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:48.289338 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281477    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281480    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281483    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281485    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281487    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281505    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281508    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281510    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281513    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281516    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281519    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281522    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281525    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281527    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281530    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281532    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281535    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281538    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281541    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:48.289823 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281543    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281546    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281549    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281551    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281554    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281557    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281559    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281562    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281565    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281568    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281570    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281573    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281576    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281580    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281583    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281586    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281589    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:48.290581 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.281592    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.281620    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.283743    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.283771    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.283777    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.283782    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.283800    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284321    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284332    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284335    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284338    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284342    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284344    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284349    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284352    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284354    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:48.291235 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284359    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284364    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284368    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284371    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284374    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284376    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284379    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284382    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284385    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284388    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284391    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284393    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284396    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284399    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284402    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284405    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284408    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284413    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284416    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284418    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:48.291638 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284421    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284423    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284426    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284431    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284434    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284436    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284439    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284442    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284444    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284447    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284450    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284453    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284455    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284458    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284461    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284463    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284466    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284469    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284472    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284474    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:48.292177 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284477    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284480    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284483    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284485    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284499    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284503    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284505    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284508    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284511    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284514    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284517    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284519    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284522    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284527    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284529    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284534    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284538    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284542    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284545    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:48.292832 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284547    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284550    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284553    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284556    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284559    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284561    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284564    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284567    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284569    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284572    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284574    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284577    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284579    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284582    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284585    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284587    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284590    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284593    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284595    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284598    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:48.293369 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284600    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284603    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284607    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284609    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284612    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284614    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284617    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284619    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284622    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284625    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284628    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284630    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284633    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.284671    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284728    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:37:48.293950 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284733    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284736    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284739    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284742    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284744    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284747    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284750    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284752    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284756    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284758    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284761    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284764    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284766    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284769    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284771    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284774    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284776    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284779    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284782    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:37:48.294307 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284784    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284787    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284790    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284792    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284795    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284799    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284803    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284806    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284808    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284811    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284813    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284816    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284819    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284821    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284824    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284827    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284829    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284832    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284834    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284837    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:37:48.294785 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284839    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284842    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284846    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284848    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284851    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284853    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284856    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284858    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284861    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284863    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284866    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284869    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284871    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284874    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284877    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284879    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284882    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284885    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284887    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284890    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:37:48.295279 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284893    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284896    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284900    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284902    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284905    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284907    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284910    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284912    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284915    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284918    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284920    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284923    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284926    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284928    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284931    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284933    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284936    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284939    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284941    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:37:48.295758 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284943    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284947    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284951    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284953    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284956    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284959    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284961    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284964    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284966    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284969    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284972    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284975    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284977    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284980    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284982    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284984    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284987    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284989    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284992    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:37:48.296193 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284994    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284997    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.284999    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.285002    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.285005    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.285007    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.285009    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:48.285012    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.285075    2570 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.287179    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.287974    2570 server.go:1014] "Starting client certificate rotation"
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.288075    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.288121    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.289901    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:37:48.296683 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.292755    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:37:48.305056 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.305038    2570 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:37:48.307920 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.307905    2570 log.go:25] "Validated CRI v1 image API"
Oct 30 18:37:48.308750 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.308727    2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:37:48.310914 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.310897    2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 87954536-a7bd-42a3-be3f-5e99711829ea:/dev/nvme0n1p3 b40beb94-29c9-4999-b75f-9814723ec024:/dev/nvme0n1p4]
Oct 30 18:37:48.310970 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.310915    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:37:48.316542 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.316418    2570 manager.go:217] Machine: {Timestamp:2026-10-30 18:37:48.315029819 +0000 UTC m=+0.295580461 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097964 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2394615c66c3df23ce5ef354fe0431 SystemUUID:ec239461-5c66-c3df-23ce-5ef354fe0431 BootID:10e447eb-f1ab-4291-bc74-174d68907b05 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:66:b8:86:5b:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:66:b8:86:5b:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:01:4b:55:a5:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:37:48.316613 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.316545    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:37:48.316641 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.316630    2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:37:48.319036 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.319015    2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:37:48.319183 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.319037    2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-68.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:37:48.319231 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.319185    2570 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:37:48.319231 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.319193    2570 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:37:48.319283 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.319234    2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:37:48.319283 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.319251    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:37:48.319946 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.319936    2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:37:48.322122 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.322111    2570 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:37:48.322231 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.322223    2570 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:37:48.322942 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.322933    2570 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:37:48.322975 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.322946    2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:37:48.322975 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.322956    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:37:48.322975 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.322965    2570 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:37:48.322975 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.322974    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:37:48.323312 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.323302    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:37:48.326015 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.325995    2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:37:48.330578 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.330564    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:37:48.331444 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.331431    2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:37:48.331485 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.331464    2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:37:48.331649 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.331638    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:37:48.331685 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.331654    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:37:48.331685 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.331661    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:37:48.331685 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.331668    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:37:48.331685 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.331675    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:37:48.331685 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.331684    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:37:48.332148 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.332140    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:37:48.332175 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.332149    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:37:48.332175 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.332157    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:37:48.332175 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.332163    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:37:48.332175 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.332171    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:37:48.332353 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.332179    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:37:48.332739 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.332719    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:37:48.332739 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.332732    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:37:48.336421 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.336405    2570 server.go:1263] "Started kubelet"
Oct 30 18:37:48.336570 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.336504    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:37:48.336655 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.336592    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:37:48.336655 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.336481    2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:37:48.337281 ip-10-0-131-68 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:37:48.337696 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.337680    2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:37:48.338696 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.338681    2570 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:37:48.340886 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.340868    2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:37:48.341834 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.341816    2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:37:48.341834 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.341823    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:37:48.342003 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.341982    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:37:48.342351 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.342332    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:37:48.342618 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.342601    2570 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:37:48.342693 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.342626    2570 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:37:48.342693 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.342664    2570 factory.go:55] Registering systemd factory
Oct 30 18:37:48.342693 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.342677    2570 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:37:48.342828 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.342741    2570 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:37:48.342828 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.342748    2570 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:37:48.343726 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.343705    2570 factory.go:153] Registering CRI-O factory
Oct 30 18:37:48.343817 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.343772    2570 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:37:48.343967 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.343821    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:37:48.344839 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.344828    2570 factory.go:103] Registering Raw factory
Oct 30 18:37:48.344901 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.344846    2570 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:37:48.345774 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.345761    2570 manager.go:319] Starting recovery of all containers
Oct 30 18:37:48.349961 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.349922    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:37:48.356572 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.356554    2570 manager.go:324] Recovery completed
Oct 30 18:37:48.362154 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.362140    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:48.363854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.363838    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:48.363923 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.363868    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:48.363923 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.363883    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:48.364427 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.364407    2570 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:37:48.364427 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.364420    2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:37:48.364545 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.364440    2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:37:48.364545 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.364455    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:48.365522 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.365507    2570 policy_none.go:50] "Start"
Oct 30 18:37:48.365580 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.365525    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:37:48.365580 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.365543    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:37:48.366101 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.366083    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:48.366173 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.366117    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:48.366173 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.366129    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:48.366912 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.366901    2570 policy_none.go:44] "Start"
Oct 30 18:37:48.395292 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.395134    2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:37:48.402484 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.402464    2570 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:37:48.402589 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.402516    2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:37:48.402589 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.402531    2570 server.go:87] "Starting device plugin registration server"
Oct 30 18:37:48.402801 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.402788    2570 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:37:48.402838 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.402806    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:37:48.402920 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.402907    2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:37:48.403022 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.402991    2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:37:48.403022 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.403000    2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:37:48.403804 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.403787    2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:37:48.403873 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.403837    2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:37:48.481232 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.481211    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:37:48.481232 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.481237    2570 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:37:48.482391 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.481257    2570 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:37:48.482391 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.481288    2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:37:48.482873 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.482578    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-68.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:48.482873 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.482595    2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-68.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:37:48.483609 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.482608    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-68.us-west-1.compute.internal.18e36294e9898b93  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-68.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-68.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 18:37:48.336376723 +0000 UTC m=+0.316927364,LastTimestamp:2026-10-30 18:37:48.336376723 +0000 UTC m=+0.316927364,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-68.us-west-1.compute.internal,}"
Oct 30 18:37:48.503540 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.503524    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:48.505031 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.505012    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:48.505109 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.505049    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:48.505109 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.505063    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:48.505109 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.505094    2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.537794 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.537629    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-68.us-west-1.compute.internal.18e36294eb2cd134  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-68.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-68.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-68.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 18:37:48.363854132 +0000 UTC m=+0.344404773,LastTimestamp:2026-10-30 18:37:48.363854132 +0000 UTC m=+0.344404773,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-68.us-west-1.compute.internal,}"
Oct 30 18:37:48.537794 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.537715    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:37:48.581886 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.581840    2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"]
Oct 30 18:37:48.582046 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.581926    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:48.583137 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.583118    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:48.583227 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.583149    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:48.583227 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.583160    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:48.584501 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.584481    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:48.584642 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.584625    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.584702 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.584664    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:48.585182 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.585164    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:48.585182 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.585177    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:48.585282 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.585195    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:48.585282 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.585201    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:48.585282 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.585207    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:48.585282 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.585217    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:48.586237 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.586222    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.586313 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.586247    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:48.586982 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.586970    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:48.587065 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.586995    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:48.587065 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.587006    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:48.588312 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.588288    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-68.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:37:48.588385 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.588307    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:37:48.607981 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.607965    2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.611739 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.611724    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:37:48.611826 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.611780    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dp942"
Oct 30 18:37:48.613702 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.613688    2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.638565 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.638459    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-68.us-west-1.compute.internal.18e36294eb2d1f1f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-68.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-68.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-68.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 18:37:48.363874079 +0000 UTC m=+0.344424720,LastTimestamp:2026-10-30 18:37:48.363874079 +0000 UTC m=+0.344424720,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-68.us-west-1.compute.internal,}"
Oct 30 18:37:48.638900 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.638881    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 18:37:48.645439 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.645423    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee243aa474c3eaea9adbefcd420da07b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"ee243aa474c3eaea9adbefcd420da07b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.645542 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.645444    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.645542 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.645464    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.692954 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.692931    2570 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-68.us-west-1.compute.internal\" not found" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.746249 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.746214    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.746249 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.746247    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.746514 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.746264    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee243aa474c3eaea9adbefcd420da07b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"ee243aa474c3eaea9adbefcd420da07b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.746514 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.746313    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.746514 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.746370    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee243aa474c3eaea9adbefcd420da07b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"ee243aa474c3eaea9adbefcd420da07b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.746514 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.746395    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.786399 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.786370    2570 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-68.us-west-1.compute.internal\" not found" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.893825 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.893740    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:48.894732 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.894713    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:48.894822 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.894742    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:48.894822 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.894752    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:48.894822 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.894778    2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.910890 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.910869    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.917191 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.917173    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.949047 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:48.949021    2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:48.949047 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:48.949049    2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-68.us-west-1.compute.internal\": node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:37:49.063238 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:49.063209    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:37:49.163628 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:49.163542    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:37:49.264036 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:49.264002    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:37:49.288181 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.288159    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:37:49.288604 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:49.288324    2570 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aec3e5e704cd849e78ca3402ce7e1bb4-9d227f372467f59d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-68.us-west-1.compute.internal.18e36294eb2cd134\": read tcp 10.0.131.68:41224->3.128.161.143:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-68.us-west-1.compute.internal.18e36294eb2cd134  default   4823 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-68.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-68.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-68.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 18:37:48 +0000 UTC,LastTimestamp:2026-10-30 18:37:48.583134147 +0000 UTC m=+0.563684788,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-68.us-west-1.compute.internal,}"
Oct 30 18:37:49.364493 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:49.364467    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:37:49.465427 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:49.465343    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:37:49.566277 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:49.566227    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:37:49.615941 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.615891    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:32:49 +0000 UTC" deadline="2028-03-28 05:03:41.775922248 +0000 UTC"
Oct 30 18:37:49.615941 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.615937    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12346h25m52.159989459s"
Oct 30 18:37:49.622034 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.622008    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:49.623046 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.623027    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:49.623104 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.623063    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:49.623104 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.623075    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:49.656012 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.655984    2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:49.743184 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.743096    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:49.806509 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.806479    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:37:49.808207 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.808188    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:37:49.871473 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.871447    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:37:49.921560 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:49.921517    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f4594c497c6167e1c46e6910eabe36.slice/crio-0fa96f5cce75764a86c8ea494f2895371b53f8b1fdd5ef2c8d552aef1aa71b15 WatchSource:0}: Error finding container 0fa96f5cce75764a86c8ea494f2895371b53f8b1fdd5ef2c8d552aef1aa71b15: Status 404 returned error can't find the container with id 0fa96f5cce75764a86c8ea494f2895371b53f8b1fdd5ef2c8d552aef1aa71b15
Oct 30 18:37:49.927396 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:49.927373    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:37:49.952791 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:49.952758    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee243aa474c3eaea9adbefcd420da07b.slice/crio-b6c1bd0f51e85516fbf275dd9b45dcd8d0661c6646f04a3ae21b934bd37266fb WatchSource:0}: Error finding container b6c1bd0f51e85516fbf275dd9b45dcd8d0661c6646f04a3ae21b934bd37266fb: Status 404 returned error can't find the container with id b6c1bd0f51e85516fbf275dd9b45dcd8d0661c6646f04a3ae21b934bd37266fb
Oct 30 18:37:50.020594 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.020498    2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:50.044704 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.044682    2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:50.161024 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.160985    2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:50.324440 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.324352    2570 apiserver.go:52] "Watching apiserver"
Oct 30 18:37:50.343582 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.343533    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:37:50.434970 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.434933    2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:37:50.435339 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.435317    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb","openshift-cluster-node-tuning-operator/tuned-wbftc","openshift-image-registry/node-ca-tw6jk","kube-system/konnectivity-agent-sn8g8","openshift-multus/network-metrics-daemon-2hsqk","kube-system/global-pull-secret-syncer-lwdl6","kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-kmkl4","openshift-multus/multus-wbgsn","openshift-dns/node-resolver-q7r6n","openshift-network-diagnostics/network-check-target-q4l7t","openshift-network-operator/iptables-alerter-vw88c","openshift-ovn-kubernetes/ovnkube-node-c75bw"]
Oct 30 18:37:50.437462 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.437436    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.438461 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.438441    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.439404 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.439384    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.440829 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.440809    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:37:50.441805 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.441787    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:50.441894 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.441861    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:37:50.443057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.443039    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.444526 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.444286    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.447918 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.445781    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:50.447918 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.445871    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:37:50.448630 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.448608    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.450076 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.450056    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.451459 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.451442    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.452754 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.452727    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:50.452840 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.452778    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:37:50.454922 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.454904    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-netns\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.454979 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.454931    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-netns\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.454979 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.454948    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-var-lib-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.454979 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.454965    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-multus\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455147 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.454981    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-kubelet\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455147 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455029    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-conf-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455147 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455082    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-multus-certs\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455147 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455118    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:50.455337 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455157    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cnibin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455337 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455183    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cni-binary-copy\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455337 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455238    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-k8s-cni-cncf-io\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455337 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455266    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-hostroot\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455337 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455294    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5728634b-06b7-41f9-a4ee-9894ef4c84b9-host\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.455535 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455344    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-etc-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455535 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-node-log\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455535 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-config\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455535 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455482    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-env-overrides\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455535 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455501    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a51d3c2-ed1d-4eef-8e78-4835629fd506-iptables-alerter-script\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.455535 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455533    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455552    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-bin\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455567    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-netd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455582    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovn-node-metrics-cert\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455596    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5728634b-06b7-41f9-a4ee-9894ef4c84b9-serviceca\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455612    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deae8a19-42d0-4d66-8029-f2f131584fbc-konnectivity-ca\") pod \"konnectivity-agent-sn8g8\" (UID: \"deae8a19-42d0-4d66-8029-f2f131584fbc\") " pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455631    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9dwn\" (UniqueName: \"kubernetes.io/projected/12820ef8-1bd5-4328-b158-b427594571af-kube-api-access-w9dwn\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455653    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-systemd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455673    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-ovn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455712    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-script-lib\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455748    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-system-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455770    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455786    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-daemon-config\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455815    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deae8a19-42d0-4d66-8029-f2f131584fbc-agent-certs\") pod \"konnectivity-agent-sn8g8\" (UID: \"deae8a19-42d0-4d66-8029-f2f131584fbc\") " pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455841    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-kubelet\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.455857 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455857    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-systemd-units\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455882    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-slash\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455909    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hbn\" (UniqueName: \"kubernetes.io/projected/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-kube-api-access-t9hbn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455933    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-bin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.455971    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlq92\" (UniqueName: \"kubernetes.io/projected/8730c894-ed9b-48f8-ba2f-8c87530bdab2-kube-api-access-mlq92\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.456033    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a51d3c2-ed1d-4eef-8e78-4835629fd506-host-slash\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.456057    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-log-socket\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.456078    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.456096    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.456120    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-socket-dir-parent\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.456140    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-etc-kubernetes\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.456172    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4k9p\" (UniqueName: \"kubernetes.io/projected/3a51d3c2-ed1d-4eef-8e78-4835629fd506-kube-api-access-d4k9p\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.456323 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.456202    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgjsl\" (UniqueName: \"kubernetes.io/projected/5728634b-06b7-41f9-a4ee-9894ef4c84b9-kube-api-access-rgjsl\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.486581 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.486533    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal" event={"ID":"ee243aa474c3eaea9adbefcd420da07b","Type":"ContainerStarted","Data":"b6c1bd0f51e85516fbf275dd9b45dcd8d0661c6646f04a3ae21b934bd37266fb"}
Oct 30 18:37:50.487866 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.487843    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal" event={"ID":"72f4594c497c6167e1c46e6910eabe36","Type":"ContainerStarted","Data":"0fa96f5cce75764a86c8ea494f2895371b53f8b1fdd5ef2c8d552aef1aa71b15"}
Oct 30 18:37:50.494106 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.493995    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:37:50.494106 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.494003    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:37:50.494106 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.493995    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:37:50.494106 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.493995    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:37:50.494106 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.494024    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:37:50.494106 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.494054    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:37:50.494472 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.494139    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nsz9w\""
Oct 30 18:37:50.494472 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.494141    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:37:50.494472 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.494209    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s4fk8\""
Oct 30 18:37:50.494880 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.494859    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:37:50.494984 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.494893    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:37:50.494984 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.494934    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:37:50.494984 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.494973    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kbv4s\""
Oct 30 18:37:50.495847 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.495825    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:37:50.495964 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.495952    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:37:50.496015 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.496004    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xlrq8\""
Oct 30 18:37:50.498604 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.498484    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:37:50.499504 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.498793    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-476j6\""
Oct 30 18:37:50.499504 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.498815    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:37:50.499504 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.498834    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:37:50.499504 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.498800    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:37:50.499504 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.499479    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:37:50.502830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.502811    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gk56f\""
Oct 30 18:37:50.502909 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.502848    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:37:50.502909 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.502850    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:37:50.502909 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.502852    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:37:50.503701 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.503686    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:37:50.545923 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.545471    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:37:50.556885 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.556853    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-host\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.556885 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.556890    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.557071 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.556910    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26k5x\" (UniqueName: \"kubernetes.io/projected/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kube-api-access-26k5x\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.557071 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.556996    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-dbus\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:50.557071 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557019    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-log-socket\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.557172 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557081    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-log-socket\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.557172 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557127    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-socket-dir-parent\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.557172 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557166    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-modprobe-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.557265 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557203    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-run\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.557265 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557221    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-tmp\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.557265 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557237    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.557368 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557261    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.557368 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557269    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-socket-dir-parent\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.557368 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557291    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-netns\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.557368 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557321    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.557368 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557339    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-sys\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.557536 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557390    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-netns\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.557536 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557415    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-kubelet-config\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:50.557536 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557435    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:50.557536 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557465    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-netns\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.557536 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557494    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-var-lib-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.557536 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557524    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-netns\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.557755 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557575    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-kubelet\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.557755 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557591    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-var-lib-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.557755 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557627    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-conf-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.557755 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557648    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.557755 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cnibin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.557755 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557720    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-conf-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.557755 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557729    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-kubelet\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.558073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557771    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cnibin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.558073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557793    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-k8s-cni-cncf-io\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.558073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557860    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-k8s-cni-cncf-io\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.558073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557873    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-hostroot\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.558073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557915    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5728634b-06b7-41f9-a4ee-9894ef4c84b9-host\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.558073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557950    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-hostroot\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.558073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557948    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wt79\" (UniqueName: \"kubernetes.io/projected/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-kube-api-access-6wt79\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.558073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.557996    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5728634b-06b7-41f9-a4ee-9894ef4c84b9-host\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.558073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558003    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-etc-tuned\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.558073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558039    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558089    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-node-log\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558147    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-env-overrides\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558182    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a51d3c2-ed1d-4eef-8e78-4835629fd506-iptables-alerter-script\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558152    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-node-log\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-netd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558230    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-netd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558248    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovn-node-metrics-cert\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558280    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5728634b-06b7-41f9-a4ee-9894ef4c84b9-serviceca\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558313    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-system-cni-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558350    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9dwn\" (UniqueName: \"kubernetes.io/projected/12820ef8-1bd5-4328-b158-b427594571af-kube-api-access-w9dwn\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558377    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-ovn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558398    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlq92\" (UniqueName: \"kubernetes.io/projected/8730c894-ed9b-48f8-ba2f-8c87530bdab2-kube-api-access-mlq92\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558414    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deae8a19-42d0-4d66-8029-f2f131584fbc-agent-certs\") pod \"konnectivity-agent-sn8g8\" (UID: \"deae8a19-42d0-4d66-8029-f2f131584fbc\") " pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558434    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-systemd-units\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558460    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-slash\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558472    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-ovn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.558511 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558482    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-bin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558524    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558606    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-systemd-units\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558660    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-slash\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558660    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-bin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558683    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19feb58-fabb-4085-8e01-e8fe3c603237-tmp-dir\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558720    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558746    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558780    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558821    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558845    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-etc-kubernetes\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558867    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjsl\" (UniqueName: \"kubernetes.io/projected/5728634b-06b7-41f9-a4ee-9894ef4c84b9-kube-api-access-rgjsl\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558886    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-lib-modules\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558902    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-conf\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558928    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-multus\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558966    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-multus-certs\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558984    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.559057 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558988    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-etc-kubernetes\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.558967    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-multus\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559002    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559029    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-var-lib-kubelet\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559035    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-multus-certs\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559061    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqtjc\" (UniqueName: \"kubernetes.io/projected/2131a561-8429-446a-a953-1235ce787378-kube-api-access-nqtjc\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559092    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-device-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559119    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f19feb58-fabb-4085-8e01-e8fe3c603237-hosts-file\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559145    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559167    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cni-binary-copy\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559202    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysconfig\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559233    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-kubernetes\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.559236    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559272    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559296    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjrm\" (UniqueName: \"kubernetes.io/projected/f19feb58-fabb-4085-8e01-e8fe3c603237-kube-api-access-sfjrm\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559322    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-etc-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559407    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-config\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559435    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.559854 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559470    2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.559543    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs podName:12820ef8-1bd5-4328-b158-b427594571af nodeName:}" failed. No retries permitted until 2026-10-30 18:37:51.059502625 +0000 UTC m=+3.040053270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs") pod "network-metrics-daemon-2hsqk" (UID: "12820ef8-1bd5-4328-b158-b427594571af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559572    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-env-overrides\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559606    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5728634b-06b7-41f9-a4ee-9894ef4c84b9-serviceca\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559630    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-etc-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559652    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-bin\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559657    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559693    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559697    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-bin\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559756    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a51d3c2-ed1d-4eef-8e78-4835629fd506-iptables-alerter-script\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559783    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deae8a19-42d0-4d66-8029-f2f131584fbc-konnectivity-ca\") pod \"konnectivity-agent-sn8g8\" (UID: \"deae8a19-42d0-4d66-8029-f2f131584fbc\") " pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559801    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-systemd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559817    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-script-lib\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559834    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-system-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559853    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559874    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-daemon-config\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559890    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a51d3c2-ed1d-4eef-8e78-4835629fd506-host-slash\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.560711 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559895    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-systemd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559907    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cnibin\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559944    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-config\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559985    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-system-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.559989    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a51d3c2-ed1d-4eef-8e78-4835629fd506-host-slash\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.560062    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.560096    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-kubelet\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.560128    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hbn\" (UniqueName: \"kubernetes.io/projected/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-kube-api-access-t9hbn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.560154    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4k9p\" (UniqueName: \"kubernetes.io/projected/3a51d3c2-ed1d-4eef-8e78-4835629fd506-kube-api-access-d4k9p\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.560194    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-systemd\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.560233    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cni-binary-copy\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.560370    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-kubelet\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.560513    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-script-lib\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.560543    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deae8a19-42d0-4d66-8029-f2f131584fbc-konnectivity-ca\") pod \"konnectivity-agent-sn8g8\" (UID: \"deae8a19-42d0-4d66-8029-f2f131584fbc\") " pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:37:50.561228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.560564    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-daemon-config\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.563432 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.563410    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovn-node-metrics-cert\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.563540 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.563524    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deae8a19-42d0-4d66-8029-f2f131584fbc-agent-certs\") pod \"konnectivity-agent-sn8g8\" (UID: \"deae8a19-42d0-4d66-8029-f2f131584fbc\") " pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:37:50.602436 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.602331    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:37:50.602436 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.602331    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:37:50.602694 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.602469    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:37:50.602694 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.602593    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:37:50.602694 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.602634    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m9qwf\""
Oct 30 18:37:50.602694 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.602646    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:37:50.602694 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.602649    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:37:50.602871 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.602725    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h468w\""
Oct 30 18:37:50.602871 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.602774    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q8mbb\""
Oct 30 18:37:50.619694 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.619653    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjsl\" (UniqueName: \"kubernetes.io/projected/5728634b-06b7-41f9-a4ee-9894ef4c84b9-kube-api-access-rgjsl\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.619842 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.619783    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlq92\" (UniqueName: \"kubernetes.io/projected/8730c894-ed9b-48f8-ba2f-8c87530bdab2-kube-api-access-mlq92\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.619896 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.619833    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9dwn\" (UniqueName: \"kubernetes.io/projected/12820ef8-1bd5-4328-b158-b427594571af-kube-api-access-w9dwn\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:50.620536 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.620520    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4k9p\" (UniqueName: \"kubernetes.io/projected/3a51d3c2-ed1d-4eef-8e78-4835629fd506-kube-api-access-d4k9p\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.621073 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.621057    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hbn\" (UniqueName: \"kubernetes.io/projected/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-kube-api-access-t9hbn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.650437 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.650404    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:37:50.650593 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.650494    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jd9zm"
Oct 30 18:37:50.661320 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661288    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-etc-tuned\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.661320 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.661539 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661344    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-system-cni-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.661539 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661363    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.661539 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661435    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19feb58-fabb-4085-8e01-e8fe3c603237-tmp-dir\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.661539 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661436    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-system-cni-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.661539 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661506    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.661539 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661438    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661484    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-lib-modules\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661563    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-conf\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661589    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661613    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661634    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-var-lib-kubelet\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661639    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-lib-modules\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtjc\" (UniqueName: \"kubernetes.io/projected/2131a561-8429-446a-a953-1235ce787378-kube-api-access-nqtjc\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661681    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-device-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661708    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f19feb58-fabb-4085-8e01-e8fe3c603237-hosts-file\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661725    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-conf\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661730    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-var-lib-kubelet\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661746    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19feb58-fabb-4085-8e01-e8fe3c603237-tmp-dir\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661784    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f19feb58-fabb-4085-8e01-e8fe3c603237-hosts-file\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661789    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-device-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.661830 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661812    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysconfig\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661852    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-kubernetes\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661862    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysconfig\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661877    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661905    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjrm\" (UniqueName: \"kubernetes.io/projected/f19feb58-fabb-4085-8e01-e8fe3c603237-kube-api-access-sfjrm\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661919    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-kubernetes\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661934    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661968    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cnibin\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.661972    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662004    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-systemd\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662014    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cnibin\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662032    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-host\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662067    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-systemd\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662078    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-host\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662096    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662125    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26k5x\" (UniqueName: \"kubernetes.io/projected/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kube-api-access-26k5x\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662150    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-dbus\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:50.662470 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662179    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-modprobe-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662204    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-run\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662269    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662272    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-tmp\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662319    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662366    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662375    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662405    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-dbus\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662414    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-run\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662367    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-modprobe-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662408    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662462    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662534    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-sys\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662521    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662577    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-kubelet-config\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662603    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-sys\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.663202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662621    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:50.663719 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662635    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-kubelet-config\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:50.663719 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662657    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.663719 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662692    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wt79\" (UniqueName: \"kubernetes.io/projected/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-kube-api-access-6wt79\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.663719 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.662703    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:50.663719 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.662757    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret podName:6bbdacee-0942-49d7-b3eb-9730db6a9a73 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:51.162743983 +0000 UTC m=+3.143294612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret") pod "global-pull-secret-syncer-lwdl6" (UID: "6bbdacee-0942-49d7-b3eb-9730db6a9a73") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:50.663719 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662845    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.663719 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.662932    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.664043 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.664024    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-etc-tuned\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.664289 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.664274    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-tmp\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.720711 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.720675    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:50.720711 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.720707    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:50.720711 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.720718    2570 projected.go:196] Error preparing data for projected volume kube-api-access-sktpg for pod openshift-network-diagnostics/network-check-target-q4l7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:50.720955 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:50.720780    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg podName:1ac5a577-b230-4d00-8d6b-270bd20c84d1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:51.220762836 +0000 UTC m=+3.201313471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sktpg" (UniqueName: "kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg") pod "network-check-target-q4l7t" (UID: "1ac5a577-b230-4d00-8d6b-270bd20c84d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:50.723102 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.723070    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wt79\" (UniqueName: \"kubernetes.io/projected/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-kube-api-access-6wt79\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.723102 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.723084    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtjc\" (UniqueName: \"kubernetes.io/projected/2131a561-8429-446a-a953-1235ce787378-kube-api-access-nqtjc\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.723482 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.723466    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26k5x\" (UniqueName: \"kubernetes.io/projected/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kube-api-access-26k5x\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.723619 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.723604    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjrm\" (UniqueName: \"kubernetes.io/projected/f19feb58-fabb-4085-8e01-e8fe3c603237-kube-api-access-sfjrm\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.750204 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.750177    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wbgsn"
Oct 30 18:37:50.757923 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.757903    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:37:50.758760 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:50.758735    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8730c894_ed9b_48f8_ba2f_8c87530bdab2.slice/crio-9888c08f7c8fb7adbe9263fa190006f9045718bfd71570e019e111ad5793d0ae WatchSource:0}: Error finding container 9888c08f7c8fb7adbe9263fa190006f9045718bfd71570e019e111ad5793d0ae: Status 404 returned error can't find the container with id 9888c08f7c8fb7adbe9263fa190006f9045718bfd71570e019e111ad5793d0ae
Oct 30 18:37:50.763750 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.763729    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:37:50.765325 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:50.765301    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a51d3c2_ed1d_4eef_8e78_4835629fd506.slice/crio-bfd779f2cdc202fb8f94cf611fa30e740c930e5f2be549c166c05ac86a895023 WatchSource:0}: Error finding container bfd779f2cdc202fb8f94cf611fa30e740c930e5f2be549c166c05ac86a895023: Status 404 returned error can't find the container with id bfd779f2cdc202fb8f94cf611fa30e740c930e5f2be549c166c05ac86a895023
Oct 30 18:37:50.769920 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.769901    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:37:50.771352 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:50.771313    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5728634b_06b7_41f9_a4ee_9894ef4c84b9.slice/crio-2af19685d36c25c9f3583abb2ec6215ce39e90a2afe9c4c0bf3ac396940697a5 WatchSource:0}: Error finding container 2af19685d36c25c9f3583abb2ec6215ce39e90a2afe9c4c0bf3ac396940697a5: Status 404 returned error can't find the container with id 2af19685d36c25c9f3583abb2ec6215ce39e90a2afe9c4c0bf3ac396940697a5
Oct 30 18:37:50.777349 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:50.777328    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeae8a19_42d0_4d66_8029_f2f131584fbc.slice/crio-d0be638f747a845ab8178aa56528b94e734babbbbdef790460f5ea4b0691634a WatchSource:0}: Error finding container d0be638f747a845ab8178aa56528b94e734babbbbdef790460f5ea4b0691634a: Status 404 returned error can't find the container with id d0be638f747a845ab8178aa56528b94e734babbbbdef790460f5ea4b0691634a
Oct 30 18:37:50.781923 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.781905    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:37:50.789461 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.789443    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:37:50.789752 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:50.789730    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abb6a27_5f39_4491_8252_e0d5cf1ac6d6.slice/crio-ede1b4e06ba7a18441dba7dfb7f02dfc6d266443a1c2e1ad035a06e284f0ce8e WatchSource:0}: Error finding container ede1b4e06ba7a18441dba7dfb7f02dfc6d266443a1c2e1ad035a06e284f0ce8e: Status 404 returned error can't find the container with id ede1b4e06ba7a18441dba7dfb7f02dfc6d266443a1c2e1ad035a06e284f0ce8e
Oct 30 18:37:50.790352 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.790334    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:37:50.797282 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:50.797259    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc298d2ee_c75d_4611_8c5d_ea9dcf9e9cdf.slice/crio-06c8491fe86ac95ba6bc087b5e1cfecc99efd161d742ba51b5d4f9703312067e WatchSource:0}: Error finding container 06c8491fe86ac95ba6bc087b5e1cfecc99efd161d742ba51b5d4f9703312067e: Status 404 returned error can't find the container with id 06c8491fe86ac95ba6bc087b5e1cfecc99efd161d742ba51b5d4f9703312067e
Oct 30 18:37:50.797886 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.797812    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:37:50.801060 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:50.801023    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:37:50.809910 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:50.809886    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19feb58_fabb_4085_8e01_e8fe3c603237.slice/crio-33d1d037b13448ee782b80e1339a14e460683e6df2550303c205386d62005c58 WatchSource:0}: Error finding container 33d1d037b13448ee782b80e1339a14e460683e6df2550303c205386d62005c58: Status 404 returned error can't find the container with id 33d1d037b13448ee782b80e1339a14e460683e6df2550303c205386d62005c58
Oct 30 18:37:50.810624 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:37:50.810598    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8473e5f2_d8a6_4df6_b977_426fbd7aee60.slice/crio-e7a53446b8a0b9c4366bcdf2a0da383a27316994b87148f14fc43873d51190bd WatchSource:0}: Error finding container e7a53446b8a0b9c4366bcdf2a0da383a27316994b87148f14fc43873d51190bd: Status 404 returned error can't find the container with id e7a53446b8a0b9c4366bcdf2a0da383a27316994b87148f14fc43873d51190bd
Oct 30 18:37:51.066489 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.066401    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:51.066668 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:51.066586    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:51.066668 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:51.066649    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs podName:12820ef8-1bd5-4328-b158-b427594571af nodeName:}" failed. No retries permitted until 2026-10-30 18:37:52.066634299 +0000 UTC m=+4.047184928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs") pod "network-metrics-daemon-2hsqk" (UID: "12820ef8-1bd5-4328-b158-b427594571af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:51.166921 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.166878    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:51.167120 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:51.167033    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:51.167120 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:51.167096    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret podName:6bbdacee-0942-49d7-b3eb-9730db6a9a73 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:52.167076689 +0000 UTC m=+4.147627323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret") pod "global-pull-secret-syncer-lwdl6" (UID: "6bbdacee-0942-49d7-b3eb-9730db6a9a73") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:51.268210 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.268173    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:51.268424 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:51.268359    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:51.268424 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:51.268383    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:51.268424 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:51.268396    2570 projected.go:196] Error preparing data for projected volume kube-api-access-sktpg for pod openshift-network-diagnostics/network-check-target-q4l7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:51.268614 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:51.268452    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg podName:1ac5a577-b230-4d00-8d6b-270bd20c84d1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:52.268438655 +0000 UTC m=+4.248989286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sktpg" (UniqueName: "kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg") pod "network-check-target-q4l7t" (UID: "1ac5a577-b230-4d00-8d6b-270bd20c84d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:51.491548 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.491444    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb" event={"ID":"8473e5f2-d8a6-4df6-b977-426fbd7aee60","Type":"ContainerStarted","Data":"e7a53446b8a0b9c4366bcdf2a0da383a27316994b87148f14fc43873d51190bd"}
Oct 30 18:37:51.492896 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.492861    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q7r6n" event={"ID":"f19feb58-fabb-4085-8e01-e8fe3c603237","Type":"ContainerStarted","Data":"33d1d037b13448ee782b80e1339a14e460683e6df2550303c205386d62005c58"}
Oct 30 18:37:51.494202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.494178    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sn8g8" event={"ID":"deae8a19-42d0-4d66-8029-f2f131584fbc","Type":"ContainerStarted","Data":"d0be638f747a845ab8178aa56528b94e734babbbbdef790460f5ea4b0691634a"}
Oct 30 18:37:51.495997 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.495962    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tw6jk" event={"ID":"5728634b-06b7-41f9-a4ee-9894ef4c84b9","Type":"ContainerStarted","Data":"2af19685d36c25c9f3583abb2ec6215ce39e90a2afe9c4c0bf3ac396940697a5"}
Oct 30 18:37:51.497678 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.497652    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vw88c" event={"ID":"3a51d3c2-ed1d-4eef-8e78-4835629fd506","Type":"ContainerStarted","Data":"bfd779f2cdc202fb8f94cf611fa30e740c930e5f2be549c166c05ac86a895023"}
Oct 30 18:37:51.499105 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.499081    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wbftc" event={"ID":"2131a561-8429-446a-a953-1235ce787378","Type":"ContainerStarted","Data":"3334cb03f8914cc3dfe748bc68f6c49117a3b1f9f8bee760c71f380512ce2c2a"}
Oct 30 18:37:51.500391 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.500368    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" event={"ID":"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf","Type":"ContainerStarted","Data":"06c8491fe86ac95ba6bc087b5e1cfecc99efd161d742ba51b5d4f9703312067e"}
Oct 30 18:37:51.501955 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.501913    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw" event={"ID":"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6","Type":"ContainerStarted","Data":"ede1b4e06ba7a18441dba7dfb7f02dfc6d266443a1c2e1ad035a06e284f0ce8e"}
Oct 30 18:37:51.503527 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.503491    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wbgsn" event={"ID":"8730c894-ed9b-48f8-ba2f-8c87530bdab2","Type":"ContainerStarted","Data":"9888c08f7c8fb7adbe9263fa190006f9045718bfd71570e019e111ad5793d0ae"}
Oct 30 18:37:51.653290 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.653240    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:51 +0000 UTC" deadline="2028-06-07 20:15:39.158908615 +0000 UTC"
Oct 30 18:37:51.653290 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:51.653286    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14065h37m47.505628463s"
Oct 30 18:37:52.076706 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:52.076664    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:52.076928 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.076885    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:52.077013 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.076958    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs podName:12820ef8-1bd5-4328-b158-b427594571af nodeName:}" failed. No retries permitted until 2026-10-30 18:37:54.076935615 +0000 UTC m=+6.057486250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs") pod "network-metrics-daemon-2hsqk" (UID: "12820ef8-1bd5-4328-b158-b427594571af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:52.177904 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:52.177841    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:52.178104 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.178066    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:52.178165 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.178138    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret podName:6bbdacee-0942-49d7-b3eb-9730db6a9a73 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:54.178117856 +0000 UTC m=+6.158668489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret") pod "global-pull-secret-syncer-lwdl6" (UID: "6bbdacee-0942-49d7-b3eb-9730db6a9a73") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:52.278605 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:52.278553    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:52.279018 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.278828    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:52.279018 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.278856    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:52.279018 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.278870    2570 projected.go:196] Error preparing data for projected volume kube-api-access-sktpg for pod openshift-network-diagnostics/network-check-target-q4l7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:52.279018 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.278931    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg podName:1ac5a577-b230-4d00-8d6b-270bd20c84d1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:54.278911967 +0000 UTC m=+6.259462599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sktpg" (UniqueName: "kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg") pod "network-check-target-q4l7t" (UID: "1ac5a577-b230-4d00-8d6b-270bd20c84d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:52.490650 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:52.489240    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:52.490650 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.489386    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:37:52.490650 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:52.489853    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:52.490650 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.489958    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:37:52.490650 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:52.490026    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:52.490650 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:52.490118    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:37:52.653788 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:52.653669    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:51 +0000 UTC" deadline="2028-07-04 02:13:50.932781532 +0000 UTC"
Oct 30 18:37:52.653788 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:52.653736    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14695h35m58.279049967s"
Oct 30 18:37:53.537009 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:53.536777    2570 generic.go:356] "Generic (PLEG): container finished" podID="72f4594c497c6167e1c46e6910eabe36" containerID="e9621a28c05e0373179720f28130b2844c4c610f94061ad5001a41e55ae07dcf" exitCode=0
Oct 30 18:37:53.537143 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:53.537036    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal" event={"ID":"72f4594c497c6167e1c46e6910eabe36","Type":"ContainerDied","Data":"e9621a28c05e0373179720f28130b2844c4c610f94061ad5001a41e55ae07dcf"}
Oct 30 18:37:54.097621 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:54.097495    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:54.098116 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.097730    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:54.098116 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.097811    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs podName:12820ef8-1bd5-4328-b158-b427594571af nodeName:}" failed. No retries permitted until 2026-10-30 18:37:58.097788194 +0000 UTC m=+10.078338826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs") pod "network-metrics-daemon-2hsqk" (UID: "12820ef8-1bd5-4328-b158-b427594571af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:54.199016 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:54.198969    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:54.199221 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.199159    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:54.199277 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.199235    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret podName:6bbdacee-0942-49d7-b3eb-9730db6a9a73 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:58.199213249 +0000 UTC m=+10.179763897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret") pod "global-pull-secret-syncer-lwdl6" (UID: "6bbdacee-0942-49d7-b3eb-9730db6a9a73") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:54.299872 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:54.299760    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:54.300053 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.300016    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:54.300053 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.300043    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:54.300175 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.300056    2570 projected.go:196] Error preparing data for projected volume kube-api-access-sktpg for pod openshift-network-diagnostics/network-check-target-q4l7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:54.300175 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.300152    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg podName:1ac5a577-b230-4d00-8d6b-270bd20c84d1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:58.300129012 +0000 UTC m=+10.280679658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sktpg" (UniqueName: "kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg") pod "network-check-target-q4l7t" (UID: "1ac5a577-b230-4d00-8d6b-270bd20c84d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:54.484677 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:54.484430    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:54.484677 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.484584    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:37:54.485033 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:54.485002    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:54.485156 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.485136    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:37:54.485216 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:54.485198    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:54.485302 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:54.485280    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:37:54.548961 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:54.548915    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal" event={"ID":"ee243aa474c3eaea9adbefcd420da07b","Type":"ContainerStarted","Data":"7d567b79d9172c25d8d1e84882552cee5df175ca7469ac37d6c76afd28c1f749"}
Oct 30 18:37:54.560081 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:54.560049    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal" event={"ID":"72f4594c497c6167e1c46e6910eabe36","Type":"ContainerStarted","Data":"bccfc136bd0949666d3ee25d6569acfa3184957077def37ab5504d41cbe8d39a"}
Oct 30 18:37:54.788628 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:54.788552    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal" podStartSLOduration=4.788530514 podStartE2EDuration="4.788530514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:54.669425792 +0000 UTC m=+6.649976447" watchObservedRunningTime="2026-10-30 18:37:54.788530514 +0000 UTC m=+6.769081170"
Oct 30 18:37:54.788776 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:54.788685    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal" podStartSLOduration=4.788676224 podStartE2EDuration="4.788676224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:54.787802078 +0000 UTC m=+6.768352733" watchObservedRunningTime="2026-10-30 18:37:54.788676224 +0000 UTC m=+6.769226879"
Oct 30 18:37:55.574828 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:55.573633    2570 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="76a8949f0bee28aceb341b10fb597198bb4e1c1514971d1cd88aa1ab0845acc4" exitCode=0
Oct 30 18:37:55.574828 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:55.574635    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" event={"ID":"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf","Type":"ContainerDied","Data":"76a8949f0bee28aceb341b10fb597198bb4e1c1514971d1cd88aa1ab0845acc4"}
Oct 30 18:37:56.484912 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:56.484875    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:56.485079 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:56.485022    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:37:56.485476 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:56.485455    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:56.485621 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:56.485563    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:37:56.485696 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:56.485658    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:56.485789 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:56.485758    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:37:56.584708 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:56.584673    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sn8g8" event={"ID":"deae8a19-42d0-4d66-8029-f2f131584fbc","Type":"ContainerStarted","Data":"71ee2cb5ca269b0e83a2096ab13ec3730e714114417204ee2396e84e784d364b"}
Oct 30 18:37:57.590469 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:57.590428    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tw6jk" event={"ID":"5728634b-06b7-41f9-a4ee-9894ef4c84b9","Type":"ContainerStarted","Data":"4ff90daaa05bfe3b140a5801bd6f81d551c00ab0ca30baa9935d6a533e1d9f73"}
Oct 30 18:37:57.594652 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:57.594604    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb" event={"ID":"8473e5f2-d8a6-4df6-b977-426fbd7aee60","Type":"ContainerStarted","Data":"ef4293e72e09f7ad1ddf591ac4f391f8a52354f79e85793a10f35fbede27fd90"}
Oct 30 18:37:57.709407 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:57.709300    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sn8g8" podStartSLOduration=3.949970022 podStartE2EDuration="8.709277418s" totalImagesPullingTime="4.759307396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:56.70416738 +0000 UTC m=+8.684718036" watchObservedRunningTime="2026-10-30 18:37:57.709277418 +0000 UTC m=+9.689828073"
Oct 30 18:37:58.130901 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:58.130810    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:58.131131 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.131007    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:58.131131 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.131076    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs podName:12820ef8-1bd5-4328-b158-b427594571af nodeName:}" failed. No retries permitted until 2026-10-30 18:38:06.131055348 +0000 UTC m=+18.111605980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs") pod "network-metrics-daemon-2hsqk" (UID: "12820ef8-1bd5-4328-b158-b427594571af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:58.231855 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:58.231812    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:58.232055 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.232004    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:58.232122 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.232073    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret podName:6bbdacee-0942-49d7-b3eb-9730db6a9a73 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:06.232051221 +0000 UTC m=+18.212601869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret") pod "global-pull-secret-syncer-lwdl6" (UID: "6bbdacee-0942-49d7-b3eb-9730db6a9a73") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:58.333199 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:58.333153    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:58.333409 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.333345    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:58.333409 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.333367    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:58.333409 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.333382    2570 projected.go:196] Error preparing data for projected volume kube-api-access-sktpg for pod openshift-network-diagnostics/network-check-target-q4l7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:58.333613 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.333447    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg podName:1ac5a577-b230-4d00-8d6b-270bd20c84d1 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:06.333427707 +0000 UTC m=+18.313978354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sktpg" (UniqueName: "kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg") pod "network-check-target-q4l7t" (UID: "1ac5a577-b230-4d00-8d6b-270bd20c84d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:58.484386 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:58.483638    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:37:58.484386 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.483780    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:37:58.484386 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:58.484102    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:37:58.484386 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.484182    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:37:58.484386 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:58.484219    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:37:58.484386 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:37:58.484261    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:37:59.577461 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:59.577072    2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:37:59.603619 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:59.603555    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wbgsn" event={"ID":"8730c894-ed9b-48f8-ba2f-8c87530bdab2","Type":"ContainerStarted","Data":"0c6e2e74a385bcd7c4ca5d75cc7dd54c46bc38b61ffeb61068ea80a94ecda07b"}
Oct 30 18:37:59.607269 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:59.607204    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb" event={"ID":"8473e5f2-d8a6-4df6-b977-426fbd7aee60","Type":"ContainerStarted","Data":"02ad7d306be0e935c0ca989220f26def27adc6139813904f11cb3828ce7dc593"}
Oct 30 18:37:59.663117 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:59.662301    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:37:59.663606 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:59.663534    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:37:59.724823 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:59.724757    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tw6jk" podStartSLOduration=5.088061324 podStartE2EDuration="10.724737625s" totalImagesPullingTime="5.636676301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:57.709663439 +0000 UTC m=+9.690214097" watchObservedRunningTime="2026-10-30 18:37:59.724737625 +0000 UTC m=+11.705288282"
Oct 30 18:37:59.845468 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:37:59.845223    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wbgsn" podStartSLOduration=2.130374974 podStartE2EDuration="10.845201804s" totalImagesPullingTime="8.71482683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:59.725553204 +0000 UTC m=+11.706103860" watchObservedRunningTime="2026-10-30 18:37:59.845201804 +0000 UTC m=+11.825752459"
Oct 30 18:38:00.413308 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.412280    2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:37:59.577089717Z","UUID":"1f00999d-d5c0-4d9c-9a43-44f94e31cf05","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:38:00.414852 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.414828    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:38:00.415009 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.414866    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:38:00.482268 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.482232    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:00.482439 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:00.482381    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:00.482823 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.482802    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:00.482941 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:00.482920    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:00.483122 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.483097    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:00.483234 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:00.483211    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:00.619864 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.619823    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q7r6n" event={"ID":"f19feb58-fabb-4085-8e01-e8fe3c603237","Type":"ContainerStarted","Data":"d31dfa3ae417b0394353b35adc7341fb5752bfd1d665bfe404bf674bac3247d1"}
Oct 30 18:38:00.627609 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.627564    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" event={"ID":"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf","Type":"ContainerStarted","Data":"d12cbcb29aff3155351a4e073da5fe4b338294f17ad817902860b3843056e68e"}
Oct 30 18:38:00.627725 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.627633    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:38:00.628202 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.628181    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:38:00.740327 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:00.740258    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q7r6n" podStartSLOduration=1.7116570530000002 podStartE2EDuration="10.740235758s" totalImagesPullingTime="9.028578705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:00.739814701 +0000 UTC m=+12.720365359" watchObservedRunningTime="2026-10-30 18:38:00.740235758 +0000 UTC m=+12.720786414"
Oct 30 18:38:01.632548 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:01.632491    2570 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="d12cbcb29aff3155351a4e073da5fe4b338294f17ad817902860b3843056e68e" exitCode=0
Oct 30 18:38:01.633275 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:01.632634    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" event={"ID":"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf","Type":"ContainerDied","Data":"d12cbcb29aff3155351a4e073da5fe4b338294f17ad817902860b3843056e68e"}
Oct 30 18:38:02.482296 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:02.482266    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:02.482433 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:02.482333    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:02.482433 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:02.482334    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:02.482530 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:02.482432    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:02.482580 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:02.482549    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:02.482647 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:02.482628    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:02.637849 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:02.637763    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb" event={"ID":"8473e5f2-d8a6-4df6-b977-426fbd7aee60","Type":"ContainerStarted","Data":"af30a9d393715871e1a314f016954407285ef97b1bef417adbb92e02f92861c7"}
Oct 30 18:38:02.639395 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:02.639362    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vw88c" event={"ID":"3a51d3c2-ed1d-4eef-8e78-4835629fd506","Type":"ContainerStarted","Data":"25e3a35afa021297076fed7bea2591bd293438857b4f93a4ffdd238ee49f775b"}
Oct 30 18:38:02.641187 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:02.641144    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wbftc" event={"ID":"2131a561-8429-446a-a953-1235ce787378","Type":"ContainerStarted","Data":"5995ae4d5c8e07c23ea3d71a265a7e8d903c3e3ac0ac6bd4ba020dfb87c66da9"}
Oct 30 18:38:02.748298 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:02.748246    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb" podStartSLOduration=3.327937677 podStartE2EDuration="13.748226909s" totalImagesPullingTime="10.420289232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:02.748048632 +0000 UTC m=+14.728599281" watchObservedRunningTime="2026-10-30 18:38:02.748226909 +0000 UTC m=+14.728777561"
Oct 30 18:38:02.857878 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:02.857807    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vw88c" podStartSLOduration=4.603277507 podStartE2EDuration="13.857785982s" totalImagesPullingTime="9.254508475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:02.857399039 +0000 UTC m=+14.837949683" watchObservedRunningTime="2026-10-30 18:38:02.857785982 +0000 UTC m=+14.838336632"
Oct 30 18:38:02.969150 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:02.968687    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wbftc" podStartSLOduration=2.483425919 podStartE2EDuration="13.968669523s" totalImagesPullingTime="11.485243604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:02.96785352 +0000 UTC m=+14.948404169" watchObservedRunningTime="2026-10-30 18:38:02.968669523 +0000 UTC m=+14.949220175"
Oct 30 18:38:03.646065 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:03.646021    2570 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="ffe01374fbcd9580dff721236d40ad638f9691d472354f62050ec7c3e522823e" exitCode=0
Oct 30 18:38:03.646588 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:03.646552    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" event={"ID":"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf","Type":"ContainerDied","Data":"ffe01374fbcd9580dff721236d40ad638f9691d472354f62050ec7c3e522823e"}
Oct 30 18:38:04.481713 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:04.481506    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:04.481887 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:04.481543    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:04.481887 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:04.481802    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:04.482006 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:04.481888    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:04.482006 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:04.481616    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:04.482006 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:04.482000    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:05.653166 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:05.653129    2570 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="ca60f356aba90f9f422db75e261f4b4bbf8e90c417c60ea07cff860a6a4f7619" exitCode=0
Oct 30 18:38:05.653618 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:05.653184    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" event={"ID":"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf","Type":"ContainerDied","Data":"ca60f356aba90f9f422db75e261f4b4bbf8e90c417c60ea07cff860a6a4f7619"}
Oct 30 18:38:06.202130 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:06.202103    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:06.202277 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.202203    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:06.202277 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.202249    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs podName:12820ef8-1bd5-4328-b158-b427594571af nodeName:}" failed. No retries permitted until 2026-10-30 18:38:22.202236249 +0000 UTC m=+34.182786877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs") pod "network-metrics-daemon-2hsqk" (UID: "12820ef8-1bd5-4328-b158-b427594571af") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:38:06.303405 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:06.303370    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:06.303602 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.303476    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:06.303602 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.303552    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret podName:6bbdacee-0942-49d7-b3eb-9730db6a9a73 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:22.303535757 +0000 UTC m=+34.284086386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret") pod "global-pull-secret-syncer-lwdl6" (UID: "6bbdacee-0942-49d7-b3eb-9730db6a9a73") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:38:06.403702 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:06.403672    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:06.403870 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.403778    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:38:06.403870 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.403791    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:38:06.403870 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.403799    2570 projected.go:196] Error preparing data for projected volume kube-api-access-sktpg for pod openshift-network-diagnostics/network-check-target-q4l7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:06.403870 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.403840    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg podName:1ac5a577-b230-4d00-8d6b-270bd20c84d1 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:22.403828163 +0000 UTC m=+34.384378791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sktpg" (UniqueName: "kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg") pod "network-check-target-q4l7t" (UID: "1ac5a577-b230-4d00-8d6b-270bd20c84d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:38:06.482194 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:06.482120    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:06.482350 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:06.482130    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:06.482350 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:06.482256    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:06.482350 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.482252    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:06.482350 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.482342    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:06.482476 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:06.482398    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:06.658305 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:06.658270    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw" event={"ID":"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6","Type":"ContainerStarted","Data":"2158b8f6c29478248d1cbdbdf0ce66c0016eacc9a9c06f915220e182ee0ee62b"}
Oct 30 18:38:07.666378 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:07.665924    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" event={"ID":"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf","Type":"ContainerStarted","Data":"89139a1384525ef05cf199a9e369fdacd65589ac182051c2dc7ff2b6372a6635"}
Oct 30 18:38:07.668948 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:07.668922    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw" event={"ID":"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6","Type":"ContainerStarted","Data":"cbf4a2a2257626b520a11c4e0d6ba5821d4bc2881379c8fd51abe995eb13ef2b"}
Oct 30 18:38:07.669039 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:07.669024    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw" event={"ID":"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6","Type":"ContainerStarted","Data":"51f0ef3f7a08cf97dfa493dc6165ff0102f4191f307ca091cdf37c4a06a3517f"}
Oct 30 18:38:07.669096 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:07.669083    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw" event={"ID":"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6","Type":"ContainerStarted","Data":"c9163d2984cbb6112e8c7f54b89cff4cf7580dccd09047315ce3f0ab2e82e8ee"}
Oct 30 18:38:07.669150 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:07.669138    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw" event={"ID":"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6","Type":"ContainerStarted","Data":"9ecca83ace4ded7e16fd434e512c2f1abded15c0efe08e639dc58cbee8c6a8a5"}
Oct 30 18:38:07.669203 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:07.669191    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw" event={"ID":"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6","Type":"ContainerStarted","Data":"4ba208d9982d074dec589921fb98b0834b9ed5678b9df6d0725146688961ef1a"}
Oct 30 18:38:08.483159 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:08.483122    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:08.483333 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:08.483249    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:08.483333 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:08.483275    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:08.483333 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:08.483307    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:08.483539 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:08.483355    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:08.483539 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:08.483429    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:08.673441 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:08.673403    2570 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="89139a1384525ef05cf199a9e369fdacd65589ac182051c2dc7ff2b6372a6635" exitCode=0
Oct 30 18:38:08.673441 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:08.673431    2570 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="e0ea51d88b8b78ee6639c3da700cb7d6e9727c32cbfa945451a16f44d1150d26" exitCode=0
Oct 30 18:38:08.673850 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:08.673460    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" event={"ID":"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf","Type":"ContainerDied","Data":"89139a1384525ef05cf199a9e369fdacd65589ac182051c2dc7ff2b6372a6635"}
Oct 30 18:38:08.673850 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:08.673510    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" event={"ID":"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf","Type":"ContainerDied","Data":"e0ea51d88b8b78ee6639c3da700cb7d6e9727c32cbfa945451a16f44d1150d26"}
Oct 30 18:38:09.679365 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:09.679325    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw" event={"ID":"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6","Type":"ContainerStarted","Data":"7a7ed96ac5a69e7bb0e6acd76940a4038c42859dbc37fec8dfe29d31becd62bf"}
Oct 30 18:38:09.682249 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:09.682226    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" event={"ID":"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf","Type":"ContainerStarted","Data":"86cbb30716ca1e4eee6263ca405bce34895587d599cabb7fb1ef0f43dbe5b1f5"}
Oct 30 18:38:09.899341 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:09.899286    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kmkl4" podStartSLOduration=7.97972091 podStartE2EDuration="20.89927347s" totalImagesPullingTime="12.91955256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:09.899135412 +0000 UTC m=+21.879686061" watchObservedRunningTime="2026-10-30 18:38:09.89927347 +0000 UTC m=+21.879824120"
Oct 30 18:38:10.481761 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:10.481724    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:10.481761 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:10.481742    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:10.481961 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:10.481749    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:10.481961 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:10.481835    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:10.481961 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:10.481902    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:10.482045 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:10.481984    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:11.689692 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:11.689451    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw" event={"ID":"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6","Type":"ContainerStarted","Data":"0770057a248a2f7537bd157aed163b3c577c795fefaefe3a199db341dd70c673"}
Oct 30 18:38:11.690225 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:11.689730    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:11.690225 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:11.689765    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:11.705315 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:11.705285    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:11.956329 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:11.956226    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw" podStartSLOduration=7.192636767 podStartE2EDuration="22.956211244s" totalImagesPullingTime="15.763574477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:11.940151285 +0000 UTC m=+23.920701934" watchObservedRunningTime="2026-10-30 18:38:11.956211244 +0000 UTC m=+23.936761891"
Oct 30 18:38:12.481936 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:12.481898    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:12.482220 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:12.481900    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:12.482220 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:12.482044    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:12.482220 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:12.481900    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:12.482220 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:12.482085    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:12.482220 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:12.482170    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:12.691760 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:12.691725    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:12.706698 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:12.706674    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:13.806447 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:13.806410    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lwdl6"]
Oct 30 18:38:13.807314 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:13.806573    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:13.807314 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:13.806676    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:13.809272 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:13.809247    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q4l7t"]
Oct 30 18:38:13.809379 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:13.809360    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:13.809482 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:13.809463    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:13.809806 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:13.809785    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2hsqk"]
Oct 30 18:38:13.809977 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:13.809962    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:13.810080 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:13.810063    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:15.481903 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:15.481866    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:15.482250 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:15.481967    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:15.482250 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:15.482074    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:15.482250 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:15.481972    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:15.482250 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:15.481973    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:15.482250 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:15.482175    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:17.481561 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:17.481528    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:17.481561 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:17.481550    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:17.482081 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:17.481642    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:17.482081 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:17.481682    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:17.482081 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:17.481740    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:17.482081 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:17.481817    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:19.481866 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:19.481688    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:19.482268 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:19.481688    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:19.482268 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:19.481939    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4l7t" podUID="1ac5a577-b230-4d00-8d6b-270bd20c84d1"
Oct 30 18:38:19.482268 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:19.481688    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:19.482268 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:19.482006    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwdl6" podUID="6bbdacee-0942-49d7-b3eb-9730db6a9a73"
Oct 30 18:38:19.482268 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:19.482132    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2hsqk" podUID="12820ef8-1bd5-4328-b158-b427594571af"
Oct 30 18:38:20.324211 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:20.324181    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeReady"
Oct 30 18:38:20.324366 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:20.324318    2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:38:21.481908 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:21.481873    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:21.482264 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:21.481990    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:21.482264 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:21.481880    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:21.534091 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:21.534054    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:38:21.583694 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:21.583662    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:38:21.583824 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:21.583704    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6nrlk\""
Oct 30 18:38:21.583824 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:21.583721    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9gxzb\""
Oct 30 18:38:21.583824 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:21.583752    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:38:21.583824 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:21.583818    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:38:21.583945 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:21.583841    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-cts7j\""
Oct 30 18:38:22.209054 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:22.209018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:22.209235 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:22.209125    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:38:22.209235 ip-10-0-131-68 kubenswrapper[2570]: E1030 18:38:22.209186    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs podName:12820ef8-1bd5-4328-b158-b427594571af nodeName:}" failed. No retries permitted until 2026-10-30 18:38:54.209173247 +0000 UTC m=+66.189723888 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs") pod "network-metrics-daemon-2hsqk" (UID: "12820ef8-1bd5-4328-b158-b427594571af") : secret "metrics-daemon-secret" not found
Oct 30 18:38:22.310228 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:22.310194    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:22.313976 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:22.313947    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:22.410899 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:22.410859    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:22.413379 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:22.413354    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:22.472512 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:22.472429    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:22.510462 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:22.510435    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:22.838728 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:22.838658    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q7r6n_f19feb58-fabb-4085-8e01-e8fe3c603237/dns-node-resolver/0.log"
Oct 30 18:38:22.909570 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:22.909542    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lwdl6"]
Oct 30 18:38:22.912509 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:38:22.912474    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbdacee_0942_49d7_b3eb_9730db6a9a73.slice/crio-af9c4db4c2a99613347c8e5295444eaca8aad665aff1c577bda9ab3672668b88 WatchSource:0}: Error finding container af9c4db4c2a99613347c8e5295444eaca8aad665aff1c577bda9ab3672668b88: Status 404 returned error can't find the container with id af9c4db4c2a99613347c8e5295444eaca8aad665aff1c577bda9ab3672668b88
Oct 30 18:38:22.959224 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:22.959202    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q4l7t"]
Oct 30 18:38:22.962261 ip-10-0-131-68 kubenswrapper[2570]: W1030 18:38:22.962229    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac5a577_b230_4d00_8d6b_270bd20c84d1.slice/crio-d9a2f29b337b6810fc9e6a6c3b1054ae60a90cf98b93b6ccce1b0a858ad0deac WatchSource:0}: Error finding container d9a2f29b337b6810fc9e6a6c3b1054ae60a90cf98b93b6ccce1b0a858ad0deac: Status 404 returned error can't find the container with id d9a2f29b337b6810fc9e6a6c3b1054ae60a90cf98b93b6ccce1b0a858ad0deac
Oct 30 18:38:23.579764 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:23.579739    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tw6jk_5728634b-06b7-41f9-a4ee-9894ef4c84b9/node-ca/0.log"
Oct 30 18:38:23.717039 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:23.717005    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lwdl6" event={"ID":"6bbdacee-0942-49d7-b3eb-9730db6a9a73","Type":"ContainerStarted","Data":"af9c4db4c2a99613347c8e5295444eaca8aad665aff1c577bda9ab3672668b88"}
Oct 30 18:38:23.718064 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:23.718040    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q4l7t" event={"ID":"1ac5a577-b230-4d00-8d6b-270bd20c84d1","Type":"ContainerStarted","Data":"d9a2f29b337b6810fc9e6a6c3b1054ae60a90cf98b93b6ccce1b0a858ad0deac"}
Oct 30 18:38:28.732213 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:28.732170    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q4l7t" event={"ID":"1ac5a577-b230-4d00-8d6b-270bd20c84d1","Type":"ContainerStarted","Data":"09e62e885c36af4161989f240b1c91e76a3e876f3a28e9d3bddc5d30ea4d26b5"}
Oct 30 18:38:28.732676 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:28.732270    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:28.843891 ip-10-0-131-68 kubenswrapper[2570]: I1030 18:38:28.843840    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q4l7t" podStartSLOduration=35.14175871 podStartE2EDuration="39.843828133s" totalImagesPullingTime="4.702069423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:28.843571626 +0000 UTC m=+40.824122276" watchObservedRunningTime="2026-10-30 18:38:28.843828133 +0000 UTC m=+40.824378782"
Oct 30 18:38:29.053657 ip-10-0-131-68 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:38:29.067904 ip-10-0-131-68 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:38:29.068079 ip-10-0-131-68 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:38:29.068128 ip-10-0-131-68 systemd[1]: kubelet.service: Consumed 2.233s CPU time.
Oct 30 18:38:29.082158 ip-10-0-131-68 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:38:29.141689 ip-10-0-131-68 kubenswrapper[4267]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:29.141689 ip-10-0-131-68 kubenswrapper[4267]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:38:29.141689 ip-10-0-131-68 kubenswrapper[4267]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:29.141689 ip-10-0-131-68 kubenswrapper[4267]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:29.144429 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144329    4267 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:29.144429 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144425    4267 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:29.144429 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144430    4267 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:29.144429 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144433    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:29.144429 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144436    4267 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144440    4267 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144443    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144445    4267 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144448    4267 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144451    4267 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144454    4267 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144456    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144459    4267 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144462    4267 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144465    4267 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144467    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144470    4267 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144473    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144475    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144478    4267 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144480    4267 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144483    4267 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144486    4267 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144502    4267 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:29.144649 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144505    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144508    4267 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144510    4267 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144513    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144516    4267 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144527    4267 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144530    4267 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144533    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144536    4267 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144538    4267 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144541    4267 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144544    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144547    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144549    4267 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144552    4267 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144555    4267 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144559    4267 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144564    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144567    4267 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144570    4267 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:29.145260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144573    4267 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144576    4267 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144579    4267 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144582    4267 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144584    4267 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144587    4267 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144590    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144593    4267 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144596    4267 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144598    4267 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144601    4267 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144603    4267 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144608    4267 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144611    4267 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144613    4267 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144616    4267 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144619    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144622    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144625    4267 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:29.145776 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144627    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144630    4267 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144633    4267 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144635    4267 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144640    4267 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144643    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144646    4267 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144649    4267 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144651    4267 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144654    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144657    4267 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144660    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144664    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144666    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144669    4267 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144672    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144675    4267 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144678    4267 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144682    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:29.146245 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144685    4267 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144687    4267 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144690    4267 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144693    4267 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144696    4267 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144698    4267 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144701    4267 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144704    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144706    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144709    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144712    4267 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144715    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144717    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144719    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144722    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144726    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144729    4267 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144732    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144734    4267 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:29.146754 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144737    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144739    4267 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144742    4267 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144744    4267 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.144747    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145558    4267 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145567    4267 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145570    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145573    4267 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145577    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145580    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145583    4267 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145585    4267 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145589    4267 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145592    4267 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145594    4267 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145597    4267 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145599    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145602    4267 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145605    4267 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:29.147260 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145608    4267 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145610    4267 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145613    4267 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145615    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145618    4267 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145620    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145624    4267 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145626    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145629    4267 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145632    4267 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145634    4267 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145637    4267 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145640    4267 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145642    4267 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145646    4267 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145649    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145652    4267 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145655    4267 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145658    4267 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145660    4267 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:29.147786 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145663    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145666    4267 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145670    4267 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145673    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145675    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145678    4267 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145681    4267 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145685    4267 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145689    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145692    4267 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145695    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145698    4267 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145700    4267 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145703    4267 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145706    4267 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145708    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145711    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145713    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145716    4267 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:29.148274 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145719    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145721    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145724    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145726    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145731    4267 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145735    4267 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145738    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145742    4267 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145746    4267 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145749    4267 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145751    4267 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145754    4267 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145757    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145760    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145762    4267 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145765    4267 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145768    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145770    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145773    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:29.148757 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145775    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145779    4267 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145782    4267 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145785    4267 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145788    4267 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145790    4267 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145793    4267 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145796    4267 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145799    4267 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145801    4267 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145804    4267 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145806    4267 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145809    4267 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145812    4267 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145814    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145817    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145819    4267 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145822    4267 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145824    4267 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:29.149254 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145827    4267 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145830    4267 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145833    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145836    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145840    4267 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145843    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145845    4267 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145848    4267 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145851    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145854    4267 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145857    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145859    4267 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145862    4267 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.145864    4267 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146412    4267 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146420    4267 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146425    4267 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146429    4267 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146432    4267 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:29.149724 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146436    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146439    4267 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146441    4267 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146444    4267 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146447    4267 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146450    4267 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146453    4267 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146455    4267 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146458    4267 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146461    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146464    4267 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146467    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146469    4267 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146473    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146475    4267 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146478    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146481    4267 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146484    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146487    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:29.150193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146508    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146511    4267 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146514    4267 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146516    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146519    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146522    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146524    4267 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146527    4267 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146530    4267 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146534    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146538    4267 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146541    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146544    4267 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146547    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146549    4267 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146552    4267 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146554    4267 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146557    4267 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146560    4267 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146562    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:29.150751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146565    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146568    4267 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146571    4267 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146574    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146576    4267 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146579    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146582    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146584    4267 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146587    4267 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146589    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146593    4267 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146595    4267 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146598    4267 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146601    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146603    4267 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146606    4267 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146608    4267 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146611    4267 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146614    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146617    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:29.151238 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146620    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146622    4267 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146626    4267 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146629    4267 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146631    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146634    4267 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146636    4267 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146639    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146641    4267 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146644    4267 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146647    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146649    4267 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146651    4267 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146654    4267 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146656    4267 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146659    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146661    4267 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146664    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146666    4267 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:29.151751 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146669    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146673    4267 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146677    4267 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146680    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146684    4267 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146691    4267 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146694    4267 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146696    4267 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146699    4267 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146701    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146704    4267 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146707    4267 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146709    4267 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146712    4267 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146715    4267 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146717    4267 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146723    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146726    4267 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146729    4267 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:29.152202 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146731    4267 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146734    4267 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146736    4267 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.146739    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146862    4267 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146877    4267 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146884    4267 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146889    4267 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146893    4267 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146896    4267 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146901    4267 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146905    4267 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146909    4267 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146912    4267 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146916    4267 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146920    4267 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146923    4267 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146926    4267 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146932    4267 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146935    4267 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146938    4267 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146941    4267 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146945    4267 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146948    4267 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:38:29.152690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146951    4267 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146954    4267 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146958    4267 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146962    4267 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146965    4267 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146968    4267 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146973    4267 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146977    4267 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146980    4267 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146983    4267 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146986    4267 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146989    4267 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146994    4267 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.146997    4267 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147000    4267 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147003    4267 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147009    4267 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147013    4267 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147017    4267 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147021    4267 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147024    4267 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147027    4267 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147030    4267 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147034    4267 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:38:29.153278 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147037    4267 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147040    4267 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147046    4267 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147049    4267 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147052    4267 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147056    4267 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147059    4267 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147062    4267 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147065    4267 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147068    4267 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147072    4267 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147076    4267 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147079    4267 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147082    4267 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147086    4267 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147089    4267 flags.go:64] FLAG: --help="false"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147092    4267 flags.go:64] FLAG: --hostname-override="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147095    4267 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147099    4267 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147102    4267 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147106    4267 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147109    4267 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147112    4267 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:38:29.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147115    4267 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147119    4267 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147123    4267 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147126    4267 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147129    4267 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147132    4267 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147135    4267 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147138    4267 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147141    4267 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147144    4267 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147147    4267 flags.go:64] FLAG: --lock-file=""
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147152    4267 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147155    4267 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147158    4267 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147164    4267 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147167    4267 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147170    4267 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147173    4267 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147176    4267 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147180    4267 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147183    4267 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147186    4267 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147190    4267 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147194    4267 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147197    4267 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:38:29.154446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147201    4267 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147204    4267 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147207    4267 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147210    4267 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147214    4267 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147217    4267 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147220    4267 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147228    4267 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147233    4267 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147236    4267 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147239    4267 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147242    4267 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147245    4267 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147248    4267 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147251    4267 flags.go:64] FLAG: --port="10250"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147254    4267 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147257    4267 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02a1af29f821041ef"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147261    4267 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147265    4267 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147269    4267 flags.go:64] FLAG: --register-node="true"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147272    4267 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147276    4267 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147279    4267 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147283    4267 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147286    4267 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147290    4267 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:38:29.155096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147293    4267 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147296    4267 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147299    4267 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147302    4267 flags.go:64] FLAG: --runonce="false"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147305    4267 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147309    4267 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147312    4267 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147316    4267 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147319    4267 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147322    4267 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147325    4267 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147329    4267 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147332    4267 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147335    4267 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147338    4267 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147341    4267 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147344    4267 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147348    4267 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147351    4267 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147356    4267 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147359    4267 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147362    4267 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147366    4267 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147369    4267 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147374    4267 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:38:29.155859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147377    4267 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147380    4267 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147391    4267 flags.go:64] FLAG: --v="2"
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147396    4267 flags.go:64] FLAG: --version="false"
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147400    4267 flags.go:64] FLAG: --vmodule=""
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147404    4267 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147408    4267 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147557    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147561    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147564    4267 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147567    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147570    4267 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147573    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147575    4267 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147578    4267 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147581    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147584    4267 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147586    4267 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147589    4267 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147592    4267 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147595    4267 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:29.156459 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147597    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147600    4267 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147603    4267 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147605    4267 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147608    4267 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147611    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147613    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147617    4267 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147620    4267 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147622    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147626    4267 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147629    4267 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147632    4267 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147635    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147637    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147640    4267 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147643    4267 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147646    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147648    4267 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:29.157026 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147653    4267 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147657    4267 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147660    4267 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147663    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147667    4267 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147670    4267 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147673    4267 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147675    4267 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147678    4267 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147681    4267 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147683    4267 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147686    4267 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147689    4267 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147691    4267 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147694    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147696    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147699    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147701    4267 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147704    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147706    4267 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:29.157701 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147709    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147712    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147715    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147719    4267 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147721    4267 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147724    4267 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147727    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147729    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147732    4267 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147735    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147738    4267 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147740    4267 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147743    4267 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147746    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147748    4267 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147751    4267 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147753    4267 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147756    4267 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147759    4267 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:29.158544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147762    4267 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147764    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147767    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147770    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147773    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147775    4267 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147778    4267 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147780    4267 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147783    4267 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147787    4267 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147791    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147794    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147797    4267 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147800    4267 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147803    4267 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147807    4267 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147810    4267 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147813    4267 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147816    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:29.159190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147818    4267 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147821    4267 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147823    4267 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147826    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147828    4267 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147831    4267 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147833    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147836    4267 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147839    4267 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147841    4267 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147844    4267 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147846    4267 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147849    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147851    4267 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147854    4267 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.147880    4267 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:29.159700 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147916    4267 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147921    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147924    4267 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147927    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147930    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147932    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147935    4267 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147938    4267 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147940    4267 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147943    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147946    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147948    4267 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147952    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147955    4267 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147958    4267 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147960    4267 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147963    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147966    4267 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147968    4267 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147971    4267 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:29.160131 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147974    4267 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147977    4267 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147979    4267 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147982    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147985    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147987    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147990    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147992    4267 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147995    4267 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.147998    4267 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148000    4267 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148004    4267 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148008    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148011    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148014    4267 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148017    4267 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148019    4267 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148022    4267 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148024    4267 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:29.160761 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148027    4267 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148030    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148033    4267 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148035    4267 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148038    4267 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148042    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148045    4267 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148048    4267 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148051    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148053    4267 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148056    4267 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148059    4267 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148061    4267 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148064    4267 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148066    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148069    4267 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148071    4267 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148074    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148077    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148079    4267 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:29.161397 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148082    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148084    4267 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148087    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148090    4267 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148092    4267 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148095    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148098    4267 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148101    4267 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148104    4267 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148107    4267 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148109    4267 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148112    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148115    4267 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148117    4267 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148120    4267 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148122    4267 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148125    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148129    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148132    4267 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148134    4267 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:29.162045 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148137    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148140    4267 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148142    4267 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148145    4267 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148147    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148150    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148152    4267 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148155    4267 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148158    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148160    4267 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148164    4267 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148168    4267 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148171    4267 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148174    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148178    4267 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148181    4267 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148184    4267 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148187    4267 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148191    4267 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:29.162652 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148193    4267 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148197    4267 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148200    4267 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148202    4267 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148205    4267 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148207    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148210    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.148212    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.152226    4267 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.152241    4267 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.152260    4267 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.152265    4267 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152352    4267 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152358    4267 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152361    4267 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152364    4267 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152367    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152370    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152373    4267 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:29.163132 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152375    4267 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152378    4267 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152381    4267 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152383    4267 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152386    4267 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152388    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152391    4267 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152394    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152396    4267 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152399    4267 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152402    4267 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152405    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152420    4267 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152424    4267 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152434    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152438    4267 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152441    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152443    4267 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152446    4267 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152450    4267 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:29.163620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152453    4267 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152455    4267 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152458    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152461    4267 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152463    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152466    4267 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152469    4267 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152471    4267 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152474    4267 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152477    4267 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152479    4267 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152482    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152484    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152487    4267 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152504    4267 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152508    4267 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152511    4267 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152513    4267 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152517    4267 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:29.164190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152522    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152525    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152528    4267 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152532    4267 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152536    4267 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152539    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152542    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152545    4267 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152554    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152557    4267 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152559    4267 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152562    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152565    4267 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152569    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152571    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152574    4267 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152576    4267 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152579    4267 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152582    4267 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:29.164658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152584    4267 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152587    4267 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152589    4267 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152592    4267 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152594    4267 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152597    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152599    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152602    4267 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152605    4267 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152607    4267 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152610    4267 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152612    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152615    4267 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152618    4267 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152620    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152623    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152626    4267 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152628    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152631    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152634    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:29.165118 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152637    4267 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152639    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152647    4267 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152650    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152653    4267 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152655    4267 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152658    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152660    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152663    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152666    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152668    4267 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152671    4267 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152673    4267 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152676    4267 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152678    4267 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152681    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152684    4267 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152686    4267 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152689    4267 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152692    4267 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:29.165619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152695    4267 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.152724    4267 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152914    4267 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152919    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152922    4267 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152926    4267 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152928    4267 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152931    4267 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152933    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152936    4267 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152939    4267 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152942    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152944    4267 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152947    4267 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152949    4267 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152959    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:29.166241 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152963    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152965    4267 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152968    4267 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152970    4267 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152973    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152975    4267 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152978    4267 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152981    4267 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152983    4267 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152986    4267 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152988    4267 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152991    4267 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152994    4267 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152996    4267 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.152999    4267 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153001    4267 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153004    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153006    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153009    4267 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153013    4267 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:29.166728 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153015    4267 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153018    4267 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153020    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153023    4267 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153025    4267 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153028    4267 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153031    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153034    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153037    4267 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153040    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153042    4267 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153045    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153053    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153056    4267 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153061    4267 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153064    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153067    4267 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153071    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153074    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:29.167388 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153077    4267 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153080    4267 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153082    4267 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153085    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153087    4267 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153090    4267 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153094    4267 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153098    4267 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153100    4267 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153103    4267 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153106    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153108    4267 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153111    4267 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153113    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153116    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153118    4267 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153121    4267 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153124    4267 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153126    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:29.168037 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153129    4267 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153135    4267 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153138    4267 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153140    4267 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153143    4267 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153145    4267 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153148    4267 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153157    4267 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153160    4267 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153163    4267 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153165    4267 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153168    4267 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153170    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153173    4267 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153176    4267 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153179    4267 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153181    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153184    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153187    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:29.168765 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153189    4267 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153192    4267 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153194    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153197    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153199    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153202    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153205    4267 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153207    4267 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153210    4267 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153212    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153215    4267 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153218    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153220    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153223    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153227    4267 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.153251    4267 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:29.169237 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153297    4267 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153301    4267 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153305    4267 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153308    4267 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153310    4267 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153319    4267 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153322    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153325    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153328    4267 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153331    4267 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153334    4267 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153337    4267 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153339    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153342    4267 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153344    4267 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153347    4267 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153350    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153353    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153355    4267 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153358    4267 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:29.169639 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153361    4267 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153363    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153366    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153369    4267 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153372    4267 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153375    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153378    4267 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153380    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153383    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153386    4267 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153389    4267 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153393    4267 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153396    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153399    4267 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153401    4267 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153405    4267 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153409    4267 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153412    4267 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153422    4267 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:29.170159 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153425    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153428    4267 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153431    4267 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153433    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153436    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153438    4267 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153441    4267 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153444    4267 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153447    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153449    4267 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153451    4267 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153454    4267 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153457    4267 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153459    4267 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153462    4267 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153464    4267 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153467    4267 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153470    4267 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153472    4267 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:29.170789 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153475    4267 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153477    4267 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153480    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153482    4267 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153485    4267 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153505    4267 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153510    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153513    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153516    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153519    4267 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153521    4267 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153524    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153527    4267 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153536    4267 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153539    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153541    4267 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153544    4267 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153547    4267 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153549    4267 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153552    4267 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:29.171323 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153555    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153557    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153560    4267 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153563    4267 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153565    4267 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153568    4267 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153571    4267 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153573    4267 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153576    4267 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153578    4267 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153581    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153584    4267 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153587    4267 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153589    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153592    4267 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153595    4267 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153597    4267 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153601    4267 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153604    4267 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153607    4267 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:29.172027 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153609    4267 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153612    4267 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153615    4267 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153617    4267 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153621    4267 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153623    4267 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153632    4267 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:29.153635    4267 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.153688    4267 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.154777    4267 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.154847    4267 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.155029    4267 server.go:1014] "Starting client certificate rotation"
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.155238    4267 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.155372    4267 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:32:49 +0000 UTC" deadline="2028-04-23 11:42:54.155378027 +0000 UTC"
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.155444    4267 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12977h4m24.999935621s"
Oct 30 18:38:29.172708 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.155669    4267 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:38:29.173310 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.157012    4267 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:38:29.173310 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.159792    4267 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:38:29.173310 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.161280    4267 log.go:25] "Validated CRI v1 image API"
Oct 30 18:38:29.173310 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.162076    4267 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:38:29.173310 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.163539    4267 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 87954536-a7bd-42a3-be3f-5e99711829ea:/dev/nvme0n1p3 b40beb94-29c9-4999-b75f-9814723ec024:/dev/nvme0n1p4]
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.163562    4267 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06c8491fe86ac95ba6bc087b5e1cfecc99efd161d742ba51b5d4f9703312067e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06c8491fe86ac95ba6bc087b5e1cfecc99efd161d742ba51b5d4f9703312067e/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fa96f5cce75764a86c8ea494f2895371b53f8b1fdd5ef2c8d552aef1aa71b15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fa96f5cce75764a86c8ea494f2895371b53f8b1fdd5ef2c8d552aef1aa71b15/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2af19685d36c25c9f3583abb2ec6215ce39e90a2afe9c4c0bf3ac396940697a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2af19685d36c25c9f3583abb2ec6215ce39e90a2afe9c4c0bf3ac396940697a5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33d1d037b13448ee782b80e1339a14e460683e6df2550303c205386d62005c58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33d1d037b13448ee782b80e1339a14e460683e6df2550303c205386d62005c58/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9888c08f7c8fb7adbe9263fa190006f9045718bfd71570e019e111ad5793d0ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9888c08f7c8fb7adbe9263fa190006f9045718bfd71570e019e111ad5793d0ae/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af9c4db4c2a99613347c8e5295444eaca8aad665aff1c577bda9ab3672668b88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af9c4db4c2a99613347c8e5295444eaca8aad665aff1c577bda9ab3672668b88/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6c1bd0f51e85516fbf275dd9b45dcd8d0661c6646f04a3ae21b934bd37266fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6c1bd0f51e85516fbf275dd9b45dcd8d0661c6646f04a3ae21b934bd37266fb/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bfd779f2cdc202fb8f94cf611fa30e740c930e5f2be549c166c05ac86a895023/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bfd779f2cdc202fb8f94cf611fa30e740c930e5f2be549c166c05ac86a895023/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0be638f747a845ab8178aa56528b94e734babbbbdef790460f5ea4b0691634a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0be638f747a845ab8178aa56528b94e734babbbbdef790460f5ea4b0691634a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9a2f29b337b6810fc9e6a6c3b1054ae60a90cf98b93b6ccce1b0a858ad0deac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9a2f29b337b6810fc9e6a6c3b1054ae60a90cf98b93b6ccce1b0a858ad0deac/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7a53446b8a0b9c4366bcdf2a0da383a27316994b87148f14fc43873d51190bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7a53446b8a0b9c4366bcdf2a0da383a27316994b87148f14fc43873d51190bd/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ede1b4e06ba7a18441dba7dfb7f02dfc6d266443a1c2e1ad035a06e284f0ce8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ede1b4e06ba7a18441dba7dfb7f02dfc6d266443a1c2e1ad035a06e284f0ce8e/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12820ef8-1bd5-4328-b158-b427594571af/volumes/kubernetes.io~projected/kube-api-access-w9dwn:{mountpoint:/var/lib/kubelet/pods/12820ef8-1bd5-4328-b158-b427594571af/volumes/kubernetes.io~projected/kube-api-access-w9dwn major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ac5a577-b230-4d00-8d6b-270bd20c84d1/volumes/kubernetes.io~projected/kube-api-access-sktpg:{mountpoint:/var/lib/kubelet/pods/1ac5a577-b230-4d00-8d6b-270bd20c84d1/volumes/kubernetes.io~projected/kube-api-access-sktpg major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2131a561-8429-446a-a953-1235ce787378/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2131a561-8429-446a-a953-1235ce787378/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2131a561-8429-446a-a953-1235ce787378/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2131a561-8429-446a-a953-1235ce787378/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2131a561-8429-446a-a953-1235ce787378/volumes/kubernetes.io~projected/kube-api-access-nqtjc:{mountpoint:/var/lib/kubelet/pods/2131a561-8429-446a-a953-1235ce787378/volumes/kubernetes.io~projected/kube-api-access-nqtjc major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a51d3c2-ed1d-4eef-8e78-4835629fd506/volumes/kubernetes.io~projected/kube-api-access-d4k9p:{mountpoint:/var/lib/kubelet/pods/3a51d3c2-ed1d-4eef-8e78-4835629fd506/volumes/kubernetes.io~projected/kube-api-access-d4k9p major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6/volumes/kubernetes.io~projected/kube-api-access-t9hbn:{mountpoint:/var/lib/kubelet/pods/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6/volumes/kubernetes.io~projected/kube-api-access-t9hbn major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5728634b-06b7-41f9-a4ee-9894ef4c84b9/volumes/kubernetes.io~projected/kube-api-access-rgjsl:{mountpoint:/var/lib/kubelet/pods/5728634b-06b7-41f9-a4ee-9894ef4c84b9/volumes/kubernetes.io~projected/kube-api-access-rgjsl major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bbdacee-0942-49d7-b3eb-9730db6a9a73/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6bbdacee-0942-49d7-b3eb-9730db6a9a73/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8473e5f2-d8a6-4df6-b977-426fbd7aee60/volumes/kubernetes.io~projected/kube-api-access-26k5x:{mountpoint:/var/lib/kubelet/pods/8473e5f2-d8a6-4df6-b977-426fbd7aee60/volumes/kubernetes.io~projected/kube-api-access-26k5x major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8730c894-ed9b-48f8-ba2f-8c87530bdab2/volumes/kubernetes.io~projected/kube-api-access-mlq92:{mountpoint:/var/lib/kubelet/pods/8730c894-ed9b-48f8-ba2f-8c87530bdab2/volumes/kubernetes.io~projected/kube-api-access-mlq92 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf/volumes/kubernetes.io~projected/kube-api-access-6wt79:{mountpoint:/var/lib/kubelet/pods/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf/volumes/kubernetes.io~projected/kube-api-access-6wt79 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deae8a19-42d0-4d66-8029-f2f131584fbc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/deae8a19-42d0-4d66-8029-f2f131584fbc/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f19feb58-fabb-4085-8e01-e8fe3c603237/volumes/kubernetes.io~projected/kube-api-access-sfjrm:{mountpoint:/var/lib/kubelet/pods/f19feb58-fabb-4085-8e01-e8fe3c603237/volumes/kubernetes.io~projected/kube-api-access-sfjrm major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/121a3cb80696183b78a98ab979319fdb4a26a9cb1f8d4b01abdb57c281fa5918/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9f2a67c7e98aca10038f5489c7b7285e5f97452b9d55e4766b69b343750486db/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e78010c3f23faae3d168ff40644c5bfbbab07c95cbad5fc374c395d6642aaa2c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/53b6f50cb18eec9cb24dbfc151a998a8aeb24d2fd90787ef234d7f4c43e85b58/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1016c43c2364d1d44faa01de9fbf8bb6b450e49a3193b9178bb437553aacf9ac/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3953b9e3aa78ab5e9f3dd088deea18e9026b10e882de4d79421bd12c7032ba07/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b3f59595fec77e2612c230e8d298966c6be2b2c77745a2b25e1e7b053d287fed/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d0487abfa4acb4466924dcf3a1744ada3096b303507af7a9336b837b0cb9babb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/46f12194383d83783b0642dc8a6096c445031bba371d06359d579a7d135047a8/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2a0ee7d13c86cae77b4f97187e62ea363a3652ead1f76e1b626a024bf8d7d516/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/ada90c746ca2a7eb11abd7ddf87cb904c7241153e7119749c8c345ee9a08b292/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/742fa4a819becdd56267f9c736a2ef2cf01fbdfeddf952055ed8634fda42636b/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7a135ca7f44cef77c68192937f0094cc48b5b98667e84d56490c5d78635678af/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4e76b465bb608be688c657391c14fc0c1a97d3b096a9f2cae1e004b5c7457ed8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bdebe942ef42cdd26aca1ef736a54d05f1921db0edee63c7c49eba4bf0a71623/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3a2fd45b93fe06c7d42b0e79c0465ab2de94c9cffd934259310d150cfc9ce635/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ae3e20677790220e79f5d28fe1e0340adf4fba39613317ef9371bbd804a5667a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/58dcd31795a9dae82c22dbc3ac9649372b92423651883038b51c8b3ebb2237e6/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/da4b772acef35ee39f661abb19d0314392be005e3736f8b1951b025d638f8d95/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/77b53665005ee2a3280f469d2b401ccb96056dfa3b7b98044303f683111c291a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/baaa8d4c78db264be659654e47f92658fb7dbb499caa67ee3eb0035ef28d95b1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/b621e3c5731b1b113f58a1fc95b0dc9cbecf4bab103dda5c299c01ecfad5130b/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/7f1bd47499c71088a07c4cadbd7fb4987e5fbf73afb2af6cc628ab49a6302746/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/b75deef6d82170708c9349f67e67b08535a7773a1f263a6ed5014175ef7d43e6/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/79d480e2ee4637b8d6cd99ea9298eb6d65d2941c6837f8bcb6f9a0d6203fa675/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/8b4ec23fa4b30b470c5edc4687ad397fe50271555a0a59060d1c994afa472868/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b34cfe81941d1f1700f6b520ed5fd9803e29452146f4ce4bc64adfd00cb06786/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/75a5c0ac4fe0b28940416ae71fcb6e9cbe10c50d839633443602d6b463bea6f9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e66b9fde794e02d926a3479462b2dfb578cd10599b2165a08a3790fb526052b8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/fef2e7dbdd07850dbc34006231ad3a89f2b6b4e7acdecc4a25c66d309ceb8537/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/49ceb9c3897f3be5443c699b3c3920a4a1a71125f3e0bd3697a9e26c41cc5c3b/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ab4ca7f0772ab61d3a336570db1480f5896e3a77e90a95012faa20aed9f46cb6/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/809e3dcaa333d6910a684d37a6bc0a9082e8e6711ab553538268a4a1631f684d/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/58d8c31b9dde531b9f796c4939a66dae4a4996167fa160befb917fa8148a469e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/10a21f2d30869e902267bc47a5e5e96d1aad26fecf7134fca0c35fe3526c1218/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.169337    4267 manager.go:217] Machine: {Timestamp:2026-10-30 18:38:29.168702867 +0000 UTC m=+0.056619947 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101087 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2394615c66c3df23ce5ef354fe0431 SystemUUID:ec239461-5c66-c3df-23ce-5ef354fe0431 BootID:10e447eb-f1ab-4291-bc74-174d68907b05 Filesystems:[{Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f19feb58-fabb-4085-8e01-e8fe3c603237/volumes/kubernetes.io~projected/kube-api-access-sfjrm DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06c8491fe86ac95ba6bc087b5e1cfecc99efd161d742ba51b5d4f9703312067e/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fa96f5cce75764a86c8ea494f2895371b53f8b1fdd5ef2c8d552aef1aa71b15/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1ac5a577-b230-4d00-8d6b-270bd20c84d1/volumes/kubernetes.io~projected/kube-api-access-sktpg DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af9c4db4c2a99613347c8e5295444eaca8aad665aff1c577bda9ab3672668b88/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8730c894-ed9b-48f8-ba2f-8c87530bdab2/volumes/kubernetes.io~projected/kube-api-access-mlq92 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6/volumes/kubernetes.io~projected/kube-api-access-t9hbn DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2131a561-8429-446a-a953-1235ce787378/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bfd779f2cdc202fb8f94cf611fa30e740c930e5f2be549c166c05ac86a895023/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6c1bd0f51e85516fbf275dd9b45dcd8d0661c6646f04a3ae21b934bd37266fb/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/2131a561-8429-446a-a953-1235ce787378/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ede1b4e06ba7a18441dba7dfb7f02dfc6d266443a1c2e1ad035a06e284f0ce8e/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/deae8a19-42d0-4d66-8029-f2f131584fbc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8473e5f2-d8a6-4df6-b977-426fbd7aee60/volumes/kubernetes.io~projected/kube-api-access-26k5x DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2af19685d36c25c9f3583abb2ec6215ce39e90a2afe9c4c0bf3ac396940697a5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0be638f747a845ab8178aa56528b94e734babbbbdef790460f5ea4b0691634a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5728634b-06b7-41f9-a4ee-9894ef4c84b9/volumes/kubernetes.io~projected/kube-api-access-rgjsl DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a51d3c2-ed1d-4eef-8e78-4835629fd506/volumes/kubernetes.io~projected/kube-api-access-d4k9p DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33d1d037b13448ee782b80e1339a14e460683e6df2550303c205386d62005c58/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9a2f29b337b6810fc9e6a6c3b1054ae60a90cf98b93b6ccce1b0a858ad0deac/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf/volumes/kubernetes.io~projected/kube-api-access-6wt79 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12820ef8-1bd5-4328-b158-b427594571af/volumes/kubernetes.io~projected/kube-api-access-w9dwn DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9888c08f7c8fb7adbe9263fa190006f9045718bfd71570e019e111ad5793d0ae/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7a53446b8a0b9c4366bcdf2a0da383a27316994b87148f14fc43873d51190bd/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2131a561-8429-446a-a953-1235ce787378/volumes/kubernetes.io~projected/kube-api-access-nqtjc DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6bbdacee-0942-49d7-b3eb-9730db6a9a73/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:af9c4db4c2a9961 MacAddress:0a:20:26:a2:bd:8f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:66:b8:86:5b:a1 Speed:0 Mtu:9001} {Name:br-int MacAddress:fe:05:67:b7:52:2c Speed:0 Mtu:8901} {Name:d9a2f29b337b681 MacAddress:b6:92:18:14:cb:0d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:66:b8:86:5b:a1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:d4:7d:d8:73:17 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:01:4b:55:a5:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.169598    4267 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.169647    4267 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.169927    4267 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.169949    4267 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-68.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170121    4267 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170128    4267 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170139    4267 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170159    4267 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170268    4267 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170513    4267 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170639    4267 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170684    4267 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:38:29.173548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170701    4267 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:38:29.174380 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170715    4267 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:38:29.174380 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170728    4267 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:38:29.174380 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.170740    4267 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:38:29.174380 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.171586    4267 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:38:29.174380 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.171601    4267 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:38:29.174380 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.172138    4267 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:38:29.175721 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.175701    4267 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:38:29.175759 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.175724    4267 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:38:29.175869 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.175855    4267 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:38:29.176151 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176135    4267 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:38:29.176192 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176182    4267 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:38:29.176312 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176301    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:38:29.176357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176320    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:38:29.176357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176331    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:38:29.176357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176340    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:38:29.176357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176348    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:38:29.176357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176357    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:38:29.176556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176366    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:38:29.176556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176376    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:38:29.176556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176386    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:38:29.176556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176395    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:38:29.176556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176406    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:38:29.176556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176421    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:38:29.176556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176454    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:38:29.176556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.176463    4267 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:38:29.179932 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.179918    4267 server.go:1263] "Started kubelet"
Oct 30 18:38:29.180602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.180089    4267 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:38:29.180602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.180117    4267 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:38:29.180602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.180186    4267 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:38:29.180602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.180544    4267 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:38:29.180602 ip-10-0-131-68 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:38:29.182932 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.181449    4267 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:38:29.183649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.183024    4267 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:38:29.183649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.183099    4267 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:38:29.183649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.183134    4267 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:51 +0000 UTC" deadline="2028-07-27 14:29:26.78565336 +0000 UTC"
Oct 30 18:38:29.183649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.183167    4267 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15259h50m57.602488953s"
Oct 30 18:38:29.183649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.183185    4267 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:38:29.183649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.183196    4267 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:38:29.183649 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:29.183197    4267 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:38:29.183649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.183525    4267 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.184527    4267 factory.go:55] Registering systemd factory
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.184546    4267 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185096    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deae8a19-42d0-4d66-8029-f2f131584fbc" volumeName="kubernetes.io/configmap/deae8a19-42d0-4d66-8029-f2f131584fbc-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185143    4267 factory.go:153] Registering CRI-O factory
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185150    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ac5a577-b230-4d00-8d6b-270bd20c84d1" volumeName="kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185169    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a51d3c2-ed1d-4eef-8e78-4835629fd506" volumeName="kubernetes.io/projected/3a51d3c2-ed1d-4eef-8e78-4835629fd506-kube-api-access-d4k9p" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185189    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3abb6a27-5f39-4491-8252-e0d5cf1ac6d6" volumeName="kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-config" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185201    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deae8a19-42d0-4d66-8029-f2f131584fbc" volumeName="kubernetes.io/secret/deae8a19-42d0-4d66-8029-f2f131584fbc-agent-certs" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185215    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f19feb58-fabb-4085-8e01-e8fe3c603237" volumeName="kubernetes.io/empty-dir/f19feb58-fabb-4085-8e01-e8fe3c603237-tmp-dir" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185228    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2131a561-8429-446a-a953-1235ce787378" volumeName="kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-tmp" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185242    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3abb6a27-5f39-4491-8252-e0d5cf1ac6d6" volumeName="kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-env-overrides" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185158    4267 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185260    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5728634b-06b7-41f9-a4ee-9894ef4c84b9" volumeName="kubernetes.io/projected/5728634b-06b7-41f9-a4ee-9894ef4c84b9-kube-api-access-rgjsl" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185274    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8730c894-ed9b-48f8-ba2f-8c87530bdab2" volumeName="kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185287    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8730c894-ed9b-48f8-ba2f-8c87530bdab2" volumeName="kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185300    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f19feb58-fabb-4085-8e01-e8fe3c603237" volumeName="kubernetes.io/projected/f19feb58-fabb-4085-8e01-e8fe3c603237-kube-api-access-sfjrm" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185314    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12820ef8-1bd5-4328-b158-b427594571af" volumeName="kubernetes.io/projected/12820ef8-1bd5-4328-b158-b427594571af-kube-api-access-w9dwn" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185328    4267 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185350    4267 factory.go:103] Registering Raw factory
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185363    4267 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185328    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2131a561-8429-446a-a953-1235ce787378" volumeName="kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-etc-tuned" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185473    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2131a561-8429-446a-a953-1235ce787378" volumeName="kubernetes.io/projected/2131a561-8429-446a-a953-1235ce787378-kube-api-access-nqtjc" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185524    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3abb6a27-5f39-4491-8252-e0d5cf1ac6d6" volumeName="kubernetes.io/secret/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185543    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5728634b-06b7-41f9-a4ee-9894ef4c84b9" volumeName="kubernetes.io/configmap/5728634b-06b7-41f9-a4ee-9894ef4c84b9-serviceca" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185557    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bbdacee-0942-49d7-b3eb-9730db6a9a73" volumeName="kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185570    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8473e5f2-d8a6-4df6-b977-426fbd7aee60" volumeName="kubernetes.io/projected/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kube-api-access-26k5x" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185601    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8730c894-ed9b-48f8-ba2f-8c87530bdab2" volumeName="kubernetes.io/projected/8730c894-ed9b-48f8-ba2f-8c87530bdab2-kube-api-access-mlq92" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185615    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a51d3c2-ed1d-4eef-8e78-4835629fd506" volumeName="kubernetes.io/configmap/3a51d3c2-ed1d-4eef-8e78-4835629fd506-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185645    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3abb6a27-5f39-4491-8252-e0d5cf1ac6d6" volumeName="kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185677    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3abb6a27-5f39-4491-8252-e0d5cf1ac6d6" volumeName="kubernetes.io/projected/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-kube-api-access-t9hbn" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185693    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" volumeName="kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185716    4267 manager.go:319] Starting recovery of all containers
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185706    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" volumeName="kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185751    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" volumeName="kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185767    4267 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" volumeName="kubernetes.io/projected/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-kube-api-access-6wt79" seLinuxMountContext=""
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185780    4267 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:38:29.188454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.185788    4267 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:38:29.211195 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.211140    4267 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:38:29.212654 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.212626    4267 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:38:29.212654 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.212652    4267 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:38:29.212853 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.212672    4267 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:38:29.212853 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:29.212719    4267 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:38:29.228190 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.228157    4267 generic.go:356] "Generic (PLEG): container finished" podID="72f4594c497c6167e1c46e6910eabe36" containerID="e9621a28c05e0373179720f28130b2844c4c610f94061ad5001a41e55ae07dcf" exitCode=0
Oct 30 18:38:29.240649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.240612    4267 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="e0ea51d88b8b78ee6639c3da700cb7d6e9727c32cbfa945451a16f44d1150d26" exitCode=0
Oct 30 18:38:29.240649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.240647    4267 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="89139a1384525ef05cf199a9e369fdacd65589ac182051c2dc7ff2b6372a6635" exitCode=0
Oct 30 18:38:29.240649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.240656    4267 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="ca60f356aba90f9f422db75e261f4b4bbf8e90c417c60ea07cff860a6a4f7619" exitCode=0
Oct 30 18:38:29.240939 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.240664    4267 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="ffe01374fbcd9580dff721236d40ad638f9691d472354f62050ec7c3e522823e" exitCode=0
Oct 30 18:38:29.240939 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.240674    4267 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="d12cbcb29aff3155351a4e073da5fe4b338294f17ad817902860b3843056e68e" exitCode=0
Oct 30 18:38:29.240939 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.240682    4267 generic.go:356] "Generic (PLEG): container finished" podID="c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf" containerID="76a8949f0bee28aceb341b10fb597198bb4e1c1514971d1cd88aa1ab0845acc4" exitCode=0
Oct 30 18:38:29.247379 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.247357    4267 manager.go:324] Recovery completed
Oct 30 18:38:29.258515 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.258481    4267 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:29.259732 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.259713    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:29.259821 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.259745    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:29.259821 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.259762    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:29.261041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261025    4267 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:38:29.261041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261037    4267 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:38:29.261041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261041    4267 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:29.261219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261053    4267 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:38:29.261219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261168    4267 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:38:29.261219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261179    4267 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:38:29.261219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261197    4267 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:38:29.261219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261202    4267 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:38:29.261219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261208    4267 policy_none.go:50] "Start"
Oct 30 18:38:29.261219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261219    4267 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:38:29.261405 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261230    4267 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:38:29.261405 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261355    4267 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:38:29.261405 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261362    4267 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:38:29.261405 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.261367    4267 policy_none.go:44] "Start"
Oct 30 18:38:29.262117 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.262101    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:29.262191 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.262133    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:29.262191 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.262147    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:29.266324 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.266293    4267 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:38:29.271591 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.271577    4267 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:38:29.271654 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:29.271604    4267 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:38:29.271654 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.271615    4267 server.go:87] "Starting device plugin registration server"
Oct 30 18:38:29.271944 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.271933    4267 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:38:29.271975 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.271947    4267 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:38:29.272089 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.272071    4267 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:38:29.272248 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.272208    4267 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:38:29.272248 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.272249    4267 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:38:29.272399 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.272256    4267 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:38:29.272399 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.272311    4267 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:38:29.272238471Z","UUID":"27540625-57a1-4abc-b1b3-66a68ec422a9","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:38:29.273409 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:29.273378    4267 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:38:29.273508 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:29.273419    4267 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-68.us-west-1.compute.internal\" not found"
Oct 30 18:38:29.273560 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.273543    4267 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:38:29.273593 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.273566    4267 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:38:29.313529 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.312937    4267 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"]
Oct 30 18:38:29.313664 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.313568    4267 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:29.314765 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.314748    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:29.314831 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.314781    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:29.314831 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.314794    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:29.315030 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.315018    4267 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:29.315194 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.315181    4267 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:29.316023 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.315991    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:29.316073 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316035    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:29.316073 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316061    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:29.316073 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316072    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:29.316157 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316040    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:29.316157 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316135    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:29.316239 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:29.316226    4267 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.316369 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316342    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal" event={"ID":"ee243aa474c3eaea9adbefcd420da07b","Type":"ContainerStarted","Data":"7d567b79d9172c25d8d1e84882552cee5df175ca7469ac37d6c76afd28c1f749"}
Oct 30 18:38:29.316403 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316376    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal" event={"ID":"ee243aa474c3eaea9adbefcd420da07b","Type":"ContainerStarted","Data":"b6c1bd0f51e85516fbf275dd9b45dcd8d0661c6646f04a3ae21b934bd37266fb"}
Oct 30 18:38:29.316431 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316390    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal" event={"ID":"72f4594c497c6167e1c46e6910eabe36","Type":"ContainerStarted","Data":"bccfc136bd0949666d3ee25d6569acfa3184957077def37ab5504d41cbe8d39a"}
Oct 30 18:38:29.316459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316430    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal" event={"ID":"72f4594c497c6167e1c46e6910eabe36","Type":"ContainerDied","Data":"e9621a28c05e0373179720f28130b2844c4c610f94061ad5001a41e55ae07dcf"}
Oct 30 18:38:29.316459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316437    4267 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:29.316459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.316444    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal" event={"ID":"72f4594c497c6167e1c46e6910eabe36","Type":"ContainerStarted","Data":"0fa96f5cce75764a86c8ea494f2895371b53f8b1fdd5ef2c8d552aef1aa71b15"}
Oct 30 18:38:29.317321 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.317299    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:29.317425 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.317329    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:29.317425 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.317340    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:29.317507 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:29.317468    4267 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-68.us-west-1.compute.internal\" not found" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.372283 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.372250    4267 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:29.373527 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.373484    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:29.373685 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.373544    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:29.373685 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.373559    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:29.373685 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.373583    4267 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.379666 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.379643    4267 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:29.379785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.379733    4267 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:38:29.379824 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.379787    4267 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:29.426533 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.426480    4267 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:29.430050 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.430028    4267 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:29.480595 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.480563    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee243aa474c3eaea9adbefcd420da07b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"ee243aa474c3eaea9adbefcd420da07b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.480595 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.480596    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.480797 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.480620    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.485755 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.485737    4267 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.528251 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.528215    4267 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-68.us-west-1.compute.internal" is owned by different node (oldNodeID="9e724c02-4d80-4855-b266-98d42b2e7378", newNodeID=""), cleaning up...
Oct 30 18:38:29.528379 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.528325    4267 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.528418 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.528391    4267 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.528989 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.528968    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 18:38:29.529091 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.528999    4267 setters.go:546] "Node became not ready" node="ip-10-0-131-68.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T18:38:29Z","lastTransitionTime":"2026-10-30T18:38:29Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 18:38:29.544352 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.544332    4267 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:29.544437 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:29.544374    4267 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.544437 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.544388    4267 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.581524 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.581445    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.581524 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.581480    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.581524 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.581514    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee243aa474c3eaea9adbefcd420da07b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"ee243aa474c3eaea9adbefcd420da07b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.581681 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.581568    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee243aa474c3eaea9adbefcd420da07b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"ee243aa474c3eaea9adbefcd420da07b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.581681 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.581571    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.581681 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.581571    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72f4594c497c6167e1c46e6910eabe36-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" (UID: \"72f4594c497c6167e1c46e6910eabe36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:29.599842 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:29.599818    4267 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:29.599958 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:29.599861    4267 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:30.171090 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.171051    4267 apiserver.go:52] "Watching apiserver"
Oct 30 18:38:30.249027 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.248995    4267 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:30.249169 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.249122    4267 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:30.323578 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.323544    4267 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:30.323578 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.323551    4267 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:30.323805 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:30.323607    4267 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:30.323805 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:30.323609    4267 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal"
Oct 30 18:38:30.323805 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.323696    4267 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:38:30.324139 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.324114    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-vw88c","kube-system/konnectivity-agent-sn8g8","openshift-multus/multus-additional-cni-plugins-kmkl4","openshift-multus/multus-wbgsn","kube-system/kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal","openshift-dns/node-resolver-q7r6n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-68.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-c75bw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb","openshift-cluster-node-tuning-operator/tuned-wbftc","openshift-network-diagnostics/network-check-target-q4l7t","kube-system/global-pull-secret-syncer-lwdl6","openshift-image-registry/node-ca-tw6jk","openshift-multus/network-metrics-daemon-2hsqk"]
Oct 30 18:38:30.325483 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.325445    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:30.378329 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.378289    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m9qwf\""
Oct 30 18:38:30.378851 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.378823    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gk56f\""
Oct 30 18:38:30.379270 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379250    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9gxzb\""
Oct 30 18:38:30.379424 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379407    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6nrlk\""
Oct 30 18:38:30.379529 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379508    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:38:30.379603 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379590    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q8mbb\""
Oct 30 18:38:30.379785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379730    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:38:30.379785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379740    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s4fk8\""
Oct 30 18:38:30.379785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379749    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:38:30.379785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379764    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nsz9w\""
Oct 30 18:38:30.379785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379769    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kbv4s\""
Oct 30 18:38:30.379785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379781    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:38:30.379785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379802    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:38:30.379785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379827    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:38:30.379785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379853    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xlrq8\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379915    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.379919    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380028    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380072    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380120    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380120    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380130    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380143    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380135    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380167    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380181    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:38:30.380389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380202    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:38:30.380862 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380206    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:38:30.380862 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380664    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:38:30.380862 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380697    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-476j6\""
Oct 30 18:38:30.380862 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380726    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:38:30.381042 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.380909    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:38:30.381146 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.381128    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:38:30.381329 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.381312    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:30.381380 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.381338    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:38:30.384355 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.384337    4267 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:38:30.386349 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386326    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.386446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386355    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-modprobe-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.386446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386390    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-system-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.386554 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386443    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-socket-dir-parent\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.386554 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386476    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wt79\" (UniqueName: \"kubernetes.io/projected/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-kube-api-access-6wt79\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.386554 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386535    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovn-node-metrics-cert\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.386674 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386574    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5728634b-06b7-41f9-a4ee-9894ef4c84b9-serviceca\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:38:30.386674 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386590    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-kubernetes\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.386674 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386634    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-device-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.386821 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386748    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-systemd-units\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.386821 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386798    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-var-lib-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.386918 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386830    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-node-log\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.386918 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386870    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4k9p\" (UniqueName: \"kubernetes.io/projected/3a51d3c2-ed1d-4eef-8e78-4835629fd506-kube-api-access-d4k9p\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:38:30.386918 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386899    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-lib-modules\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.387041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386923    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cni-binary-copy\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.387041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386936    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovn-node-metrics-cert\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.387041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386948    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-netns\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.387041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.386970    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-etc-kubernetes\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.387041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387009    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-system-cni-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.387243 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387063    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cnibin\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.387243 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387097    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.387243 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387158    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-ovn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.387243 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387185    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.387243 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387210    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deae8a19-42d0-4d66-8029-f2f131584fbc-konnectivity-ca\") pod \"konnectivity-agent-sn8g8\" (UID: \"deae8a19-42d0-4d66-8029-f2f131584fbc\") " pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:38:30.387243 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387234    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-netns\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.387519 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387267    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-log-socket\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.387519 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387271    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cni-binary-copy\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.387519 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387326    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5728634b-06b7-41f9-a4ee-9894ef4c84b9-serviceca\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:38:30.387519 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387330    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-bin\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.387519 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387348    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.387519 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387367    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-config\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.387519 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387421    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-env-overrides\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.387519 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387449    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deae8a19-42d0-4d66-8029-f2f131584fbc-konnectivity-ca\") pod \"konnectivity-agent-sn8g8\" (UID: \"deae8a19-42d0-4d66-8029-f2f131584fbc\") " pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:38:30.387799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387523    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a51d3c2-ed1d-4eef-8e78-4835629fd506-iptables-alerter-script\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:38:30.387799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387554    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-sys\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.387799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387573    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-var-lib-kubelet\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.387799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387571    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-config\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.387799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387607    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-daemon-config\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.387799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387638    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-env-overrides\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.387799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387666    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.387799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387771    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f19feb58-fabb-4085-8e01-e8fe3c603237-hosts-file\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:38:30.387799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387788    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-daemon-config\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.387799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387799    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysconfig\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.388054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387821    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-tmp\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.388054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387855    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-multus-certs\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.388054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387888    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26k5x\" (UniqueName: \"kubernetes.io/projected/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kube-api-access-26k5x\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.388054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387890    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-tmp\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.388054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387912    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.388054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387929    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-systemd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.388054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387957    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-netd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.388054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387973    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjrm\" (UniqueName: \"kubernetes.io/projected/f19feb58-fabb-4085-8e01-e8fe3c603237-kube-api-access-sfjrm\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:38:30.388054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.387994    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.388054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388021    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5728634b-06b7-41f9-a4ee-9894ef4c84b9-host\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:38:30.388357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388092    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-systemd\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.388357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388132    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-host\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.388357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388162    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cnibin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.388357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388184    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-conf-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.388357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388220    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9dwn\" (UniqueName: \"kubernetes.io/projected/12820ef8-1bd5-4328-b158-b427594571af-kube-api-access-w9dwn\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:30.388357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388251    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-kubelet\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.388357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388265    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-kubelet-config\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:30.388357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388289    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:30.388357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388329    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19feb58-fabb-4085-8e01-e8fe3c603237-tmp-dir\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:38:30.388606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388361    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-k8s-cni-cncf-io\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.388606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388399    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f19feb58-fabb-4085-8e01-e8fe3c603237-tmp-dir\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:38:30.388606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388427    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.388606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388444    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-script-lib\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.388606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388581    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-dbus\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:30.388606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388605    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-run\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.388779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388613    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-ovnkube-script-lib\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.388779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388623    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-etc-tuned\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.388779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388649    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-etc-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.388779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388675    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2131a561-8429-446a-a953-1235ce787378-etc-tuned\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.388779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388681    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.388779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388709    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hbn\" (UniqueName: \"kubernetes.io/projected/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-kube-api-access-t9hbn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.388779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388737    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtjc\" (UniqueName: \"kubernetes.io/projected/2131a561-8429-446a-a953-1235ce787378-kube-api-access-nqtjc\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.388779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388752    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:30.388779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388770    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-hostroot\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.389104 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388792    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.389104 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388828    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-multus\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.389104 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388863    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.389104 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388899    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-slash\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.389104 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.388988    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjsl\" (UniqueName: \"kubernetes.io/projected/5728634b-06b7-41f9-a4ee-9894ef4c84b9-kube-api-access-rgjsl\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:38:30.389104 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389024    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a51d3c2-ed1d-4eef-8e78-4835629fd506-host-slash\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:38:30.389104 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389053    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-kubelet\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.389104 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389099    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlq92\" (UniqueName: \"kubernetes.io/projected/8730c894-ed9b-48f8-ba2f-8c87530bdab2-kube-api-access-mlq92\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.389470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389105    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.389470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389138    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.389470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389164    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.389470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389188    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-conf\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.389470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389223    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-bin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.389470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389244    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.389470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389263    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deae8a19-42d0-4d66-8029-f2f131584fbc-agent-certs\") pod \"konnectivity-agent-sn8g8\" (UID: \"deae8a19-42d0-4d66-8029-f2f131584fbc\") " pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:38:30.389470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389334    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.389470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389365    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.389470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389407    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:30.389768 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.389473    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deae8a19-42d0-4d66-8029-f2f131584fbc-agent-certs\") pod \"konnectivity-agent-sn8g8\" (UID: \"deae8a19-42d0-4d66-8029-f2f131584fbc\") " pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:38:30.442929 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.442900    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wt79\" (UniqueName: \"kubernetes.io/projected/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-kube-api-access-6wt79\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.443085 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.442985    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9dwn\" (UniqueName: \"kubernetes.io/projected/12820ef8-1bd5-4328-b158-b427594571af-kube-api-access-w9dwn\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:30.443121 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.443112    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26k5x\" (UniqueName: \"kubernetes.io/projected/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kube-api-access-26k5x\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.444513 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.444478    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hbn\" (UniqueName: \"kubernetes.io/projected/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-kube-api-access-t9hbn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.444989 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.444973    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtjc\" (UniqueName: \"kubernetes.io/projected/2131a561-8429-446a-a953-1235ce787378-kube-api-access-nqtjc\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.445042 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.445028    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktpg\" (UniqueName: \"kubernetes.io/projected/1ac5a577-b230-4d00-8d6b-270bd20c84d1-kube-api-access-sktpg\") pod \"network-check-target-q4l7t\" (UID: \"1ac5a577-b230-4d00-8d6b-270bd20c84d1\") " pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:30.445250 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.445234    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjsl\" (UniqueName: \"kubernetes.io/projected/5728634b-06b7-41f9-a4ee-9894ef4c84b9-kube-api-access-rgjsl\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:38:30.445476 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.445461    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlq92\" (UniqueName: \"kubernetes.io/projected/8730c894-ed9b-48f8-ba2f-8c87530bdab2-kube-api-access-mlq92\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.451126 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.451104    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-cts7j\""
Oct 30 18:38:30.451239 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.451193    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h468w\""
Oct 30 18:38:30.451283 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.451268    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:38:30.460814 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.460794    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbdacee-0942-49d7-b3eb-9730db6a9a73-original-pull-secret\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:30.479743 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.479719    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:38:30.479824 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.479782    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:38:30.479824 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.479805    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:30.479876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.479857    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:38:30.479904 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.479887    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:38:30.483355 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.483337    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4k9p\" (UniqueName: \"kubernetes.io/projected/3a51d3c2-ed1d-4eef-8e78-4835629fd506-kube-api-access-d4k9p\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:38:30.483415 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.483338    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjrm\" (UniqueName: \"kubernetes.io/projected/f19feb58-fabb-4085-8e01-e8fe3c603237-kube-api-access-sfjrm\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:38:30.488241 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.488223    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a51d3c2-ed1d-4eef-8e78-4835629fd506-iptables-alerter-script\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:38:30.490400 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490382    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-ovn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.490445 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490410    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.490445 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490427    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-netns\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.490445 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490441    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-log-socket\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.490573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490455    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-bin\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.490573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490479    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-sys\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.490573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490508    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-log-socket\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.490573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490481    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-ovn\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.490573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490533    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-var-lib-kubelet\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.490573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490527    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-bin\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.490573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490539    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.490573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490558    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.490573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490558    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-netns\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.490573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490574    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f19feb58-fabb-4085-8e01-e8fe3c603237-hosts-file\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490558    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-sys\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490585    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-var-lib-kubelet\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490589    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysconfig\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490612    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-multus-certs\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490637    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f19feb58-fabb-4085-8e01-e8fe3c603237-hosts-file\") pod \"node-resolver-q7r6n\" (UID: \"f19feb58-fabb-4085-8e01-e8fe3c603237\") " pod="openshift-dns/node-resolver-q7r6n"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490582    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490679    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490692    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysconfig\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490727    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-multus-certs\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490756    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490755    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-systemd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490780    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-netd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490794    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-run-systemd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490832    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490872    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-cni-netd\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490876    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5728634b-06b7-41f9-a4ee-9894ef4c84b9-host\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490901    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5728634b-06b7-41f9-a4ee-9894ef4c84b9-host\") pod \"node-ca-tw6jk\" (UID: \"5728634b-06b7-41f9-a4ee-9894ef4c84b9\") " pod="openshift-image-registry/node-ca-tw6jk"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490903    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-systemd\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490874    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-sys-fs\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490926    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-host\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490965    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-host\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490970    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cnibin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490998    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-conf-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.490968    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-systemd\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491026    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-kubelet\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491051    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-conf-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491002    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-cnibin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491073    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-kubelet-config\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491080    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-kubelet\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491099    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491128    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-kubelet-config\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491128    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-k8s-cni-cncf-io\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491149    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491158    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-k8s-cni-cncf-io\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491164    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-dbus\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491190    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-socket-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.491161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491192    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-run\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491223    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6bbdacee-0942-49d7-b3eb-9730db6a9a73-dbus\") pod \"global-pull-secret-syncer-lwdl6\" (UID: \"6bbdacee-0942-49d7-b3eb-9730db6a9a73\") " pod="kube-system/global-pull-secret-syncer-lwdl6"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491235    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-etc-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491252    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-run\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491268    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-etc-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491282    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491311    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-hostroot\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491337    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491358    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491360    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-multus\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491368    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-hostroot\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491404    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-multus\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491409    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-etc-selinux\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491397    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-slash\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491438    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-slash\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491457    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a51d3c2-ed1d-4eef-8e78-4835629fd506-host-slash\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491488    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-kubelet\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491512    4267 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491526    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a51d3c2-ed1d-4eef-8e78-4835629fd506-host-slash\") pod \"iptables-alerter-vw88c\" (UID: \"3a51d3c2-ed1d-4eef-8e78-4835629fd506\") " pod="openshift-network-operator/iptables-alerter-vw88c"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491532    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491553    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-kubelet\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491555    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-conf\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491578    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-bin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491601    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491638    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-var-lib-cni-bin\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491662    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-registration-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491671    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491694    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491701    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491709    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-modprobe-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491664    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491724    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-system-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491722    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-sysctl-conf\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491747    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-socket-dir-parent\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491766    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-kubernetes\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491782    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-device-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491783    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-system-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491797    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-systemd-units\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491805    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-socket-dir-parent\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491820    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-modprobe-d\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491823    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-var-lib-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491764    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-multus-cni-dir\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491834    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-etc-kubernetes\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491853    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-var-lib-openvswitch\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491855    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8473e5f2-d8a6-4df6-b977-426fbd7aee60-device-dir\") pod \"aws-ebs-csi-driver-node-nlkzb\" (UID: \"8473e5f2-d8a6-4df6-b977-426fbd7aee60\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nlkzb"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491862    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-node-log\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491852    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-systemd-units\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491912    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-lib-modules\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491939    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-netns\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491950    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3abb6a27-5f39-4491-8252-e0d5cf1ac6d6-node-log\") pod \"ovnkube-node-c75bw\" (UID: \"3abb6a27-5f39-4491-8252-e0d5cf1ac6d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491965    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-etc-kubernetes\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491991    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-system-cni-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.492008    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-etc-kubernetes\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.492009    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2131a561-8429-446a-a953-1235ce787378-lib-modules\") pod \"tuned-wbftc\" (UID: \"2131a561-8429-446a-a953-1235ce787378\") " pod="openshift-cluster-node-tuning-operator/tuned-wbftc"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.492015    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cnibin\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.491986    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8730c894-ed9b-48f8-ba2f-8c87530bdab2-host-run-netns\") pod \"multus-wbgsn\" (UID: \"8730c894-ed9b-48f8-ba2f-8c87530bdab2\") " pod="openshift-multus/multus-wbgsn"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.492089    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-system-cni-dir\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.492540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.492101    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf-cnibin\") pod \"multus-additional-cni-plugins-kmkl4\" (UID: \"c298d2ee-c75d-4611-8c5d-ea9dcf9e9cdf\") " pod="openshift-multus/multus-additional-cni-plugins-kmkl4"
Oct 30 18:38:30.494125 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.493587    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12820ef8-1bd5-4328-b158-b427594571af-metrics-certs\") pod \"network-metrics-daemon-2hsqk\" (UID: \"12820ef8-1bd5-4328-b158-b427594571af\") " pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:30.625872 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:30.625829    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2hsqk"
Oct 30 18:38:31.017029 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:31.016950    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2hsqk"]
Oct 30 18:38:31.021282 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:31.021241    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12820ef8_1bd5_4328_b158_b427594571af.slice/crio-8a68dd8deac0dcc31e3765556213c1e14376e946d2c63b0d67a19155b36eb7b1 WatchSource:0}: Error finding container 8a68dd8deac0dcc31e3765556213c1e14376e946d2c63b0d67a19155b36eb7b1: Status 404 returned error can't find the container with id 8a68dd8deac0dcc31e3765556213c1e14376e946d2c63b0d67a19155b36eb7b1
Oct 30 18:38:31.024423 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:31.024403    4267 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:38:31.252193 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:31.252154    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hsqk" event={"ID":"12820ef8-1bd5-4328-b158-b427594571af","Type":"ContainerStarted","Data":"8a68dd8deac0dcc31e3765556213c1e14376e946d2c63b0d67a19155b36eb7b1"}
Oct 30 18:38:31.414156 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:31.414077    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sn8g8"
Oct 30 18:38:33.253482 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:33.253449    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:33.258452 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:33.258424    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hsqk" event={"ID":"12820ef8-1bd5-4328-b158-b427594571af","Type":"ContainerStarted","Data":"72e77dd33390ff3f88ab5e45d6c59be32c3a8cda7defb23c5a43f87b532af66d"}
Oct 30 18:38:33.258615 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:33.258471    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2hsqk" event={"ID":"12820ef8-1bd5-4328-b158-b427594571af","Type":"ContainerStarted","Data":"a34fe8f78246e2a3df7f758a3ae09f1f17fabd64dd8564b324f2c92f0a1d08c7"}
Oct 30 18:38:38.345967 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:38.345915    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:38:39.688160 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:39.688125    4267 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-68.us-west-1.compute.internal" event="NodeReady"
Oct 30 18:38:39.907704 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:39.907673    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:38:39.907877 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:39.907865    4267 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:38:40.808230 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.808197    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59"]
Oct 30 18:38:40.810002 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.809985    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59"
Oct 30 18:38:40.821427 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.821403    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-b8fk2"]
Oct 30 18:38:40.823247 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.823226    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-b8fk2"
Oct 30 18:38:40.845009 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.844978    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2km4w"]
Oct 30 18:38:40.848167 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.848145    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:40.848566 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.848540    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986fe640-11d4-4f32-bba7-2c79ccd6067e-tmp\") pod \"downloads-6dd789c8b7-b8fk2\" (UID: \"986fe640-11d4-4f32-bba7-2c79ccd6067e\") " pod="openshift-console/downloads-6dd789c8b7-b8fk2"
Oct 30 18:38:40.848648 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.848596    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2l4k\" (UniqueName: \"kubernetes.io/projected/986fe640-11d4-4f32-bba7-2c79ccd6067e-kube-api-access-j2l4k\") pod \"downloads-6dd789c8b7-b8fk2\" (UID: \"986fe640-11d4-4f32-bba7-2c79ccd6067e\") " pod="openshift-console/downloads-6dd789c8b7-b8fk2"
Oct 30 18:38:40.848696 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.848676    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/904bb0c9-c485-4f03-81f1-a0c885b2b8bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-p4g59\" (UID: \"904bb0c9-c485-4f03-81f1-a0c885b2b8bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59"
Oct 30 18:38:40.870376 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.870227    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 18:38:40.870376 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.870317    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2z5fq\""
Oct 30 18:38:40.877662 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.877641    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 18:38:40.877826 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.877807    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 18:38:40.891659 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.891632    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zk6cm"]
Oct 30 18:38:40.893619 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.893600    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:40.898392 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.898370    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c68fcb657-cq76w"]
Oct 30 18:38:40.900013 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.899994    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:40.919613 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.919588    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:38:40.919817 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.919802    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:38:40.919868 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.919854    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:38:40.936879 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.936839    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-b8fk2"]
Oct 30 18:38:40.946063 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.946040    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:38:40.949275 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949255    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2l4k\" (UniqueName: \"kubernetes.io/projected/986fe640-11d4-4f32-bba7-2c79ccd6067e-kube-api-access-j2l4k\") pod \"downloads-6dd789c8b7-b8fk2\" (UID: \"986fe640-11d4-4f32-bba7-2c79ccd6067e\") " pod="openshift-console/downloads-6dd789c8b7-b8fk2"
Oct 30 18:38:40.949343 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949288    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-metrics-tls\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:40.949343 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949307    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9467a5a9-c8a6-4952-9447-1f25ae1e2617-ca-trust-extracted\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:40.949343 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949324    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9467a5a9-c8a6-4952-9447-1f25ae1e2617-trusted-ca\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:40.949487 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949388    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/904bb0c9-c485-4f03-81f1-a0c885b2b8bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-p4g59\" (UID: \"904bb0c9-c485-4f03-81f1-a0c885b2b8bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59"
Oct 30 18:38:40.949487 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949437    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-config-volume\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:40.949579 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949554    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9467a5a9-c8a6-4952-9447-1f25ae1e2617-bound-sa-token\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:40.949630 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949595    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:40.949686 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949635    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9467a5a9-c8a6-4952-9447-1f25ae1e2617-registry-tls\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:40.949732 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949713    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcw9v\" (UniqueName: \"kubernetes.io/projected/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-kube-api-access-bcw9v\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:40.949777 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949740    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986fe640-11d4-4f32-bba7-2c79ccd6067e-tmp\") pod \"downloads-6dd789c8b7-b8fk2\" (UID: \"986fe640-11d4-4f32-bba7-2c79ccd6067e\") " pod="openshift-console/downloads-6dd789c8b7-b8fk2"
Oct 30 18:38:40.949777 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949765    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-tmp-dir\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:40.949873 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949799    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:40.949873 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949821    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9467a5a9-c8a6-4952-9447-1f25ae1e2617-registry-certificates\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:40.949873 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949837    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9lq6\" (UniqueName: \"kubernetes.io/projected/9467a5a9-c8a6-4952-9447-1f25ae1e2617-kube-api-access-j9lq6\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:40.949873 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949861    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-crio-socket\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:40.950070 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949904    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-data-volume\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:40.950070 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949938    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9467a5a9-c8a6-4952-9447-1f25ae1e2617-image-registry-private-configuration\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:40.950070 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949966    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92z5\" (UniqueName: \"kubernetes.io/projected/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-kube-api-access-j92z5\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:40.950070 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.949982    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9467a5a9-c8a6-4952-9447-1f25ae1e2617-installation-pull-secrets\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:40.950239 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.950145    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/986fe640-11d4-4f32-bba7-2c79ccd6067e-tmp\") pod \"downloads-6dd789c8b7-b8fk2\" (UID: \"986fe640-11d4-4f32-bba7-2c79ccd6067e\") " pod="openshift-console/downloads-6dd789c8b7-b8fk2"
Oct 30 18:38:40.951889 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:40.951866    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/904bb0c9-c485-4f03-81f1-a0c885b2b8bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-p4g59\" (UID: \"904bb0c9-c485-4f03-81f1-a0c885b2b8bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59"
Oct 30 18:38:41.016915 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.016842    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:38:41.016915 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.016908    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:38:41.017162 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.016917    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bhm78\""
Oct 30 18:38:41.017162 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.017014    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2529f\""
Oct 30 18:38:41.017162 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.017092    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:38:41.017162 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.017123    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:38:41.017341 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.017312    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:38:41.017402 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.017344    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-wm6z8\""
Oct 30 18:38:41.017455 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.017443    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhgds\""
Oct 30 18:38:41.017455 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.017483    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:38:41.017761 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.017738    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zk6cm"]
Oct 30 18:38:41.018135 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.018120    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2km4w"]
Oct 30 18:38:41.018441 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.018430    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gcb6r"]
Oct 30 18:38:41.019737 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.019718    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2l4k\" (UniqueName: \"kubernetes.io/projected/986fe640-11d4-4f32-bba7-2c79ccd6067e-kube-api-access-j2l4k\") pod \"downloads-6dd789c8b7-b8fk2\" (UID: \"986fe640-11d4-4f32-bba7-2c79ccd6067e\") " pod="openshift-console/downloads-6dd789c8b7-b8fk2"
Oct 30 18:38:41.024346 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.024330    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59"]
Oct 30 18:38:41.024424 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.024414    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gcb6r"
Oct 30 18:38:41.041444 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.041418    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c68fcb657-cq76w"]
Oct 30 18:38:41.050397 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050373    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-tmp-dir\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050400    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050420    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9467a5a9-c8a6-4952-9447-1f25ae1e2617-registry-certificates\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050554    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9lq6\" (UniqueName: \"kubernetes.io/projected/9467a5a9-c8a6-4952-9447-1f25ae1e2617-kube-api-access-j9lq6\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050589    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88de84e-f668-49b9-b958-8da367b4c511-cert\") pod \"ingress-canary-gcb6r\" (UID: \"f88de84e-f668-49b9-b958-8da367b4c511\") " pod="openshift-ingress-canary/ingress-canary-gcb6r"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050620    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flsjj\" (UniqueName: \"kubernetes.io/projected/f88de84e-f668-49b9-b958-8da367b4c511-kube-api-access-flsjj\") pod \"ingress-canary-gcb6r\" (UID: \"f88de84e-f668-49b9-b958-8da367b4c511\") " pod="openshift-ingress-canary/ingress-canary-gcb6r"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050656    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-crio-socket\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050678    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-data-volume\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050702    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9467a5a9-c8a6-4952-9447-1f25ae1e2617-image-registry-private-configuration\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050737    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92z5\" (UniqueName: \"kubernetes.io/projected/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-kube-api-access-j92z5\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050759    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9467a5a9-c8a6-4952-9447-1f25ae1e2617-installation-pull-secrets\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050798    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-metrics-tls\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050806    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-tmp-dir\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050821    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9467a5a9-c8a6-4952-9447-1f25ae1e2617-ca-trust-extracted\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050861    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9467a5a9-c8a6-4952-9447-1f25ae1e2617-trusted-ca\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050910    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-config-volume\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050934    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9467a5a9-c8a6-4952-9447-1f25ae1e2617-bound-sa-token\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050973    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.050996    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9467a5a9-c8a6-4952-9447-1f25ae1e2617-registry-tls\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.051027    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcw9v\" (UniqueName: \"kubernetes.io/projected/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-kube-api-access-bcw9v\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.051140    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-data-volume\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.051256    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-crio-socket\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.050535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.051336    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gcb6r"]
Oct 30 18:38:41.052609 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.052527    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-config-volume\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:41.052715 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.052697    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.053818 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.053774    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9467a5a9-c8a6-4952-9447-1f25ae1e2617-ca-trust-extracted\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.053918 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.053811    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.054036 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.054014    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9467a5a9-c8a6-4952-9447-1f25ae1e2617-image-registry-private-configuration\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.054403 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.054374    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-metrics-tls\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:41.054470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.054419    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9467a5a9-c8a6-4952-9447-1f25ae1e2617-installation-pull-secrets\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.054621 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.054594    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9467a5a9-c8a6-4952-9447-1f25ae1e2617-registry-certificates\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.054886 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.054868    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9467a5a9-c8a6-4952-9447-1f25ae1e2617-trusted-ca\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.055942 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.055919    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9467a5a9-c8a6-4952-9447-1f25ae1e2617-registry-tls\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.081678 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.081598    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:38:41.081678 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.081610    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:38:41.081678 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.081651    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:38:41.082045 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.082029    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dx4x5\""
Oct 30 18:38:41.108127 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.108097    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9lq6\" (UniqueName: \"kubernetes.io/projected/9467a5a9-c8a6-4952-9447-1f25ae1e2617-kube-api-access-j9lq6\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.108127 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.108117    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92z5\" (UniqueName: \"kubernetes.io/projected/0222cd69-3c96-4c58-b9f3-a45693ffd4fa-kube-api-access-j92z5\") pod \"dns-default-zk6cm\" (UID: \"0222cd69-3c96-4c58-b9f3-a45693ffd4fa\") " pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:41.108280 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.108184    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcw9v\" (UniqueName: \"kubernetes.io/projected/3ee82b54-0298-4100-b13a-97b6d5ea3b1d-kube-api-access-bcw9v\") pod \"insights-runtime-extractor-2km4w\" (UID: \"3ee82b54-0298-4100-b13a-97b6d5ea3b1d\") " pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.110981 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.110956    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9467a5a9-c8a6-4952-9447-1f25ae1e2617-bound-sa-token\") pod \"image-registry-7c68fcb657-cq76w\" (UID: \"9467a5a9-c8a6-4952-9447-1f25ae1e2617\") " pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.120895 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.120876    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59"
Oct 30 18:38:41.132118 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.132089    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-b8fk2"
Oct 30 18:38:41.151923 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.151885    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88de84e-f668-49b9-b958-8da367b4c511-cert\") pod \"ingress-canary-gcb6r\" (UID: \"f88de84e-f668-49b9-b958-8da367b4c511\") " pod="openshift-ingress-canary/ingress-canary-gcb6r"
Oct 30 18:38:41.151923 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.151920    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flsjj\" (UniqueName: \"kubernetes.io/projected/f88de84e-f668-49b9-b958-8da367b4c511-kube-api-access-flsjj\") pod \"ingress-canary-gcb6r\" (UID: \"f88de84e-f668-49b9-b958-8da367b4c511\") " pod="openshift-ingress-canary/ingress-canary-gcb6r"
Oct 30 18:38:41.154204 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.154170    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88de84e-f668-49b9-b958-8da367b4c511-cert\") pod \"ingress-canary-gcb6r\" (UID: \"f88de84e-f668-49b9-b958-8da367b4c511\") " pod="openshift-ingress-canary/ingress-canary-gcb6r"
Oct 30 18:38:41.159369 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.159343    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2km4w"
Oct 30 18:38:41.204692 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.204642    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:41.210821 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.210649    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:41.214719 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.214205    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flsjj\" (UniqueName: \"kubernetes.io/projected/f88de84e-f668-49b9-b958-8da367b4c511-kube-api-access-flsjj\") pod \"ingress-canary-gcb6r\" (UID: \"f88de84e-f668-49b9-b958-8da367b4c511\") " pod="openshift-ingress-canary/ingress-canary-gcb6r"
Oct 30 18:38:41.333671 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.333585    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gcb6r"
Oct 30 18:38:41.496899 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.496868    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59"]
Oct 30 18:38:41.499969 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:41.499947    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904bb0c9_c485_4f03_81f1_a0c885b2b8bc.slice/crio-c452fd0e82926fea3b047985186de1a53f14018a33c833e2367f2a50cfac0a31 WatchSource:0}: Error finding container c452fd0e82926fea3b047985186de1a53f14018a33c833e2367f2a50cfac0a31: Status 404 returned error can't find the container with id c452fd0e82926fea3b047985186de1a53f14018a33c833e2367f2a50cfac0a31
Oct 30 18:38:41.547436 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.547361    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-b8fk2"]
Oct 30 18:38:41.550705 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:41.550675    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986fe640_11d4_4f32_bba7_2c79ccd6067e.slice/crio-947ca2968d22b6f531a9661ff1e9953973ff4aa9154d9e42bd480dfe66dab139 WatchSource:0}: Error finding container 947ca2968d22b6f531a9661ff1e9953973ff4aa9154d9e42bd480dfe66dab139: Status 404 returned error can't find the container with id 947ca2968d22b6f531a9661ff1e9953973ff4aa9154d9e42bd480dfe66dab139
Oct 30 18:38:41.579138 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.579103    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6599b687f8-sp2zz"]
Oct 30 18:38:41.581603 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.581580    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.603273 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.603247    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zk6cm"]
Oct 30 18:38:41.606312 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:41.606280    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0222cd69_3c96_4c58_b9f3_a45693ffd4fa.slice/crio-3358f56256efa39adc153e48d0ce81e64bed6fad507987e918aacfaa27bd045c WatchSource:0}: Error finding container 3358f56256efa39adc153e48d0ce81e64bed6fad507987e918aacfaa27bd045c: Status 404 returned error can't find the container with id 3358f56256efa39adc153e48d0ce81e64bed6fad507987e918aacfaa27bd045c
Oct 30 18:38:41.607189 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.607161    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2km4w"]
Oct 30 18:38:41.607302 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.607228    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c68fcb657-cq76w"]
Oct 30 18:38:41.609703 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:41.609681    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee82b54_0298_4100_b13a_97b6d5ea3b1d.slice/crio-af774717fafc8dbe6758a3caedf4cecd7a06a2a309e076f52c00300dcd694dcf WatchSource:0}: Error finding container af774717fafc8dbe6758a3caedf4cecd7a06a2a309e076f52c00300dcd694dcf: Status 404 returned error can't find the container with id af774717fafc8dbe6758a3caedf4cecd7a06a2a309e076f52c00300dcd694dcf
Oct 30 18:38:41.610369 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:41.610352    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9467a5a9_c8a6_4952_9447_1f25ae1e2617.slice/crio-5ebb7698b08e7a2be34942e21d987db03f271551be92b8eff26de7d4e0c60771 WatchSource:0}: Error finding container 5ebb7698b08e7a2be34942e21d987db03f271551be92b8eff26de7d4e0c60771: Status 404 returned error can't find the container with id 5ebb7698b08e7a2be34942e21d987db03f271551be92b8eff26de7d4e0c60771
Oct 30 18:38:41.634158 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.634134    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 18:38:41.634158 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.634150    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 18:38:41.634326 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.634163    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 18:38:41.634326 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.634310    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 18:38:41.634421 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.634335    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wph9t\""
Oct 30 18:38:41.634421 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.634375    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 18:38:41.641372 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.641353    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6599b687f8-sp2zz"]
Oct 30 18:38:41.655820 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.655797    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-serving-cert\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.655928 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.655840    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-tmp\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.655928 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.655872    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-oauth-serving-cert\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.657561 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.657536    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-oauth-config\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.657648 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.657573    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmj7n\" (UniqueName: \"kubernetes.io/projected/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-kube-api-access-hmj7n\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.657648 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.657590    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-config\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.657648 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.657611    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-service-ca\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.698978 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.698847    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gcb6r"]
Oct 30 18:38:41.701576 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:41.701542    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88de84e_f668_49b9_b958_8da367b4c511.slice/crio-c440c0ec939c9de46248c273da2b0ac3e34e3e4989040b60f161202acbad9269 WatchSource:0}: Error finding container c440c0ec939c9de46248c273da2b0ac3e34e3e4989040b60f161202acbad9269: Status 404 returned error can't find the container with id c440c0ec939c9de46248c273da2b0ac3e34e3e4989040b60f161202acbad9269
Oct 30 18:38:41.758457 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.758422    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-tmp\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.758661 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.758465    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-oauth-serving-cert\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.758661 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.758514    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-oauth-config\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.758661 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.758531    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmj7n\" (UniqueName: \"kubernetes.io/projected/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-kube-api-access-hmj7n\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.758661 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.758547    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-config\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.758661 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.758566    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-service-ca\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.758661 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.758618    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-serving-cert\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.759244 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.759204    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-tmp\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.759364 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.759342    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-oauth-serving-cert\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.759451 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.759357    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-service-ca\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.759451 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.759404    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-config\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.760963 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.760944    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-serving-cert\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.761052 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.761037    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-oauth-config\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.815549 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.815521    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmj7n\" (UniqueName: \"kubernetes.io/projected/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-kube-api-access-hmj7n\") pod \"console-6599b687f8-sp2zz\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") " pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:41.891817 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:41.891714    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:42.255155 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.255122    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6599b687f8-sp2zz"]
Oct 30 18:38:42.258399 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:42.258372    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10d3e5b_debe_4c43_8eb5_c4be7e89a05d.slice/crio-12765229e79153ba72a3977f1b8d30341a9e65fc5d1ab992f5e7bdb4358633cf WatchSource:0}: Error finding container 12765229e79153ba72a3977f1b8d30341a9e65fc5d1ab992f5e7bdb4358633cf: Status 404 returned error can't find the container with id 12765229e79153ba72a3977f1b8d30341a9e65fc5d1ab992f5e7bdb4358633cf
Oct 30 18:38:42.284077 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.284039    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2km4w" event={"ID":"3ee82b54-0298-4100-b13a-97b6d5ea3b1d","Type":"ContainerStarted","Data":"a078a2560a08d1c494578774187958c919dc9a7de876bec8d641979f2d7e7970"}
Oct 30 18:38:42.284228 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.284099    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2km4w" event={"ID":"3ee82b54-0298-4100-b13a-97b6d5ea3b1d","Type":"ContainerStarted","Data":"af774717fafc8dbe6758a3caedf4cecd7a06a2a309e076f52c00300dcd694dcf"}
Oct 30 18:38:42.285099 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.285072    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gcb6r" event={"ID":"f88de84e-f668-49b9-b958-8da367b4c511","Type":"ContainerStarted","Data":"c440c0ec939c9de46248c273da2b0ac3e34e3e4989040b60f161202acbad9269"}
Oct 30 18:38:42.286196 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.286171    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-b8fk2" event={"ID":"986fe640-11d4-4f32-bba7-2c79ccd6067e","Type":"ContainerStarted","Data":"947ca2968d22b6f531a9661ff1e9953973ff4aa9154d9e42bd480dfe66dab139"}
Oct 30 18:38:42.287266 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.287243    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6599b687f8-sp2zz" event={"ID":"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d","Type":"ContainerStarted","Data":"12765229e79153ba72a3977f1b8d30341a9e65fc5d1ab992f5e7bdb4358633cf"}
Oct 30 18:38:42.288397 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.288372    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zk6cm" event={"ID":"0222cd69-3c96-4c58-b9f3-a45693ffd4fa","Type":"ContainerStarted","Data":"3358f56256efa39adc153e48d0ce81e64bed6fad507987e918aacfaa27bd045c"}
Oct 30 18:38:42.289838 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.289816    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c68fcb657-cq76w" event={"ID":"9467a5a9-c8a6-4952-9447-1f25ae1e2617","Type":"ContainerStarted","Data":"caf727a1e227a51d0505acc6766d5b3c2781a3ae8100cbbca5061555899e56e8"}
Oct 30 18:38:42.289903 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.289859    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c68fcb657-cq76w" event={"ID":"9467a5a9-c8a6-4952-9447-1f25ae1e2617","Type":"ContainerStarted","Data":"5ebb7698b08e7a2be34942e21d987db03f271551be92b8eff26de7d4e0c60771"}
Oct 30 18:38:42.290087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.290065    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:38:42.290980 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.290963    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59" event={"ID":"904bb0c9-c485-4f03-81f1-a0c885b2b8bc","Type":"ContainerStarted","Data":"c452fd0e82926fea3b047985186de1a53f14018a33c833e2367f2a50cfac0a31"}
Oct 30 18:38:42.406014 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:42.405960    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c68fcb657-cq76w" podStartSLOduration=2.405941082 podStartE2EDuration="2.405941082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:42.405591742 +0000 UTC m=+13.293508829" watchObservedRunningTime="2026-10-30 18:38:42.405941082 +0000 UTC m=+13.293858165"
Oct 30 18:38:43.297900 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:43.297850    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2km4w" event={"ID":"3ee82b54-0298-4100-b13a-97b6d5ea3b1d","Type":"ContainerStarted","Data":"51c81a7380e26636d05145ffa5f20f4d8aee2be71d18c2f6281b0fd8a897ff59"}
Oct 30 18:38:44.311191 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:44.311156    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59" event={"ID":"904bb0c9-c485-4f03-81f1-a0c885b2b8bc","Type":"ContainerStarted","Data":"09f8a222e31f43a0fa80b84d060c4235ba896723d70da664d33a3e2c695cd33a"}
Oct 30 18:38:44.312560 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:44.312171    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59"
Oct 30 18:38:44.318967 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:44.318945    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59"
Oct 30 18:38:44.542039 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:44.541970    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p4g59" podStartSLOduration=2.506528223 podStartE2EDuration="4.541949486s" totalImagesPullingTime="2.035421263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:44.423643721 +0000 UTC m=+15.311560813" watchObservedRunningTime="2026-10-30 18:38:44.541949486 +0000 UTC m=+15.429866579"
Oct 30 18:38:45.319583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.319485    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zk6cm" event={"ID":"0222cd69-3c96-4c58-b9f3-a45693ffd4fa","Type":"ContainerStarted","Data":"dc9bdaffa150c3a1bd36848e75137a0b34e9fa69e3547f9b4a061ae5820b8e67"}
Oct 30 18:38:45.319583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.319578    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:45.320073 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.319609    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zk6cm" event={"ID":"0222cd69-3c96-4c58-b9f3-a45693ffd4fa","Type":"ContainerStarted","Data":"b5e497e4d6968101406b11795bfd5ffcdc8b99af01078aefd426ad6b8c128433"}
Oct 30 18:38:45.434241 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.434165    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zk6cm" podStartSLOduration=2.417057133 podStartE2EDuration="5.434148082s" totalImagesPullingTime="3.017090949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:45.433585769 +0000 UTC m=+16.321502859" watchObservedRunningTime="2026-10-30 18:38:45.434148082 +0000 UTC m=+16.322065171"
Oct 30 18:38:45.762738 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.761754    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"]
Oct 30 18:38:45.764698 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.763876    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.793764 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.793564    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9db20512-6277-4144-8065-2f7af76decd2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.793764 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.793606    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9db20512-6277-4144-8065-2f7af76decd2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.793764 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.793626    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9db20512-6277-4144-8065-2f7af76decd2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.793764 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.793643    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmpzf\" (UniqueName: \"kubernetes.io/projected/9db20512-6277-4144-8065-2f7af76decd2-kube-api-access-vmpzf\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.817500 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.817462    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b9tfc\""
Oct 30 18:38:45.817838 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.817817    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 18:38:45.817955 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.817874    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:38:45.817955 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.817953    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:38:45.818082 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.818065    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 18:38:45.818170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.818131    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:38:45.823785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.823705    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"]
Oct 30 18:38:45.894633 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.894601    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9db20512-6277-4144-8065-2f7af76decd2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.894794 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.894639    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9db20512-6277-4144-8065-2f7af76decd2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.894794 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.894665    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9db20512-6277-4144-8065-2f7af76decd2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.894794 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.894685    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmpzf\" (UniqueName: \"kubernetes.io/projected/9db20512-6277-4144-8065-2f7af76decd2-kube-api-access-vmpzf\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.895024 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:45.894997    4267 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Oct 30 18:38:45.895162 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:45.895140    4267 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db20512-6277-4144-8065-2f7af76decd2-prometheus-operator-tls podName:9db20512-6277-4144-8065-2f7af76decd2 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:46.395115891 +0000 UTC m=+17.283032977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/9db20512-6277-4144-8065-2f7af76decd2-prometheus-operator-tls") pod "prometheus-operator-9cd89b74c-s9xdm" (UID: "9db20512-6277-4144-8065-2f7af76decd2") : secret "prometheus-operator-tls" not found
Oct 30 18:38:45.895358 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.895331    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9db20512-6277-4144-8065-2f7af76decd2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.898169 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.898141    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9db20512-6277-4144-8065-2f7af76decd2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:45.952713 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:45.952677    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmpzf\" (UniqueName: \"kubernetes.io/projected/9db20512-6277-4144-8065-2f7af76decd2-kube-api-access-vmpzf\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:46.326305 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:46.326266    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2km4w" event={"ID":"3ee82b54-0298-4100-b13a-97b6d5ea3b1d","Type":"ContainerStarted","Data":"3f2abb0167e651e95345e1dd10f7809a04f7409d4eaf6c1cbd4ae886826aec72"}
Oct 30 18:38:46.329255 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:46.329222    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gcb6r" event={"ID":"f88de84e-f668-49b9-b958-8da367b4c511","Type":"ContainerStarted","Data":"f936ef955982aa1df1ad0282386ad7961bdeb6286f2e59ba9c9dd3ea9576510f"}
Oct 30 18:38:46.400036 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:46.399842    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9db20512-6277-4144-8065-2f7af76decd2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:46.409043 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:46.409009    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9db20512-6277-4144-8065-2f7af76decd2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-s9xdm\" (UID: \"9db20512-6277-4144-8065-2f7af76decd2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:46.442873 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:46.442409    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2km4w" podStartSLOduration=2.720026652 podStartE2EDuration="6.442391611s" totalImagesPullingTime="3.722364959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:46.441027144 +0000 UTC m=+17.328944235" watchObservedRunningTime="2026-10-30 18:38:46.442391611 +0000 UTC m=+17.330308697"
Oct 30 18:38:46.554948 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:46.554018    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gcb6r" podStartSLOduration=2.136836277 podStartE2EDuration="6.554000948s" totalImagesPullingTime="4.417164671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:46.553937018 +0000 UTC m=+17.441854108" watchObservedRunningTime="2026-10-30 18:38:46.554000948 +0000 UTC m=+17.441918038"
Oct 30 18:38:46.678077 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:46.677992    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"
Oct 30 18:38:47.061966 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:47.061932    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm"]
Oct 30 18:38:47.099644 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:47.099610    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db20512_6277_4144_8065_2f7af76decd2.slice/crio-b401eaf5bad0c519b735afea0ff662c69d71943873b5e3108341b1583951ad9d WatchSource:0}: Error finding container b401eaf5bad0c519b735afea0ff662c69d71943873b5e3108341b1583951ad9d: Status 404 returned error can't find the container with id b401eaf5bad0c519b735afea0ff662c69d71943873b5e3108341b1583951ad9d
Oct 30 18:38:47.334734 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:47.334648    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6599b687f8-sp2zz" event={"ID":"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d","Type":"ContainerStarted","Data":"1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be"}
Oct 30 18:38:47.336170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:47.336139    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm" event={"ID":"9db20512-6277-4144-8065-2f7af76decd2","Type":"ContainerStarted","Data":"b401eaf5bad0c519b735afea0ff662c69d71943873b5e3108341b1583951ad9d"}
Oct 30 18:38:47.450589 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:47.450516    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6599b687f8-sp2zz" podStartSLOduration=1.7589812660000002 podStartE2EDuration="6.450484374s" totalImagesPullingTime="4.691503108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:47.450009553 +0000 UTC m=+18.337926642" watchObservedRunningTime="2026-10-30 18:38:47.450484374 +0000 UTC m=+18.338401465"
Oct 30 18:38:49.352320 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:49.352282    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm" event={"ID":"9db20512-6277-4144-8065-2f7af76decd2","Type":"ContainerStarted","Data":"567a1298a26ae0d1e604ac50dfa44418f145a7a2f971e7aedd50ee5b34ee8e19"}
Oct 30 18:38:49.352817 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:49.352340    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm" event={"ID":"9db20512-6277-4144-8065-2f7af76decd2","Type":"ContainerStarted","Data":"c207f84a9ae18322ff3cc8759e1e3fc690d71014daad2cbbc0e5c7f125ba529a"}
Oct 30 18:38:49.467447 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:49.466900    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-s9xdm" podStartSLOduration=2.663824214 podStartE2EDuration="4.466881846s" totalImagesPullingTime="1.803057632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:49.466219573 +0000 UTC m=+20.354136663" watchObservedRunningTime="2026-10-30 18:38:49.466881846 +0000 UTC m=+20.354798937"
Oct 30 18:38:50.164234 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:38:50.164196    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:38:50.791597 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:50.791561    4267 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sn8g8_deae8a19-42d0-4d66-8029-f2f131584fbc/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:51.892607 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:51.892568    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:51.893053 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:51.892662    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:38:51.894162 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:51.894132    4267 patch_prober.go:29] interesting pod/console-6599b687f8-sp2zz container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.12:8443/health\": dial tcp 10.133.0.12:8443: connect: connection refused" start-of-body=
Oct 30 18:38:51.894285 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:51.894177    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-console/console-6599b687f8-sp2zz" podUID="a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" containerName="console" probeResult="failure" output="Get \"https://10.133.0.12:8443/health\": dial tcp 10.133.0.12:8443: connect: connection refused"
Oct 30 18:38:52.472477 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.472442    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s2xhg"]
Oct 30 18:38:52.474773 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.474749    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.544614 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.544559    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/761f0730-7b45-4b34-92c9-460d84a7f2be-sys\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.544775 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.544622    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/761f0730-7b45-4b34-92c9-460d84a7f2be-root\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.544775 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.544699    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-accelerators-collector-config\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.544775 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.544761    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.544938 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.544798    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-wtmp\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.544938 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.544820    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-tls\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.544938 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.544890    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-textfile\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.545080 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.544941    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/761f0730-7b45-4b34-92c9-460d84a7f2be-metrics-client-ca\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.545080 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.544964    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndqh\" (UniqueName: \"kubernetes.io/projected/761f0730-7b45-4b34-92c9-460d84a7f2be-kube-api-access-fndqh\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.578928 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.578900    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:38:52.578928 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.578923    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:38:52.579112 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.578933    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:38:52.579112 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.578964    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xqkxc\""
Oct 30 18:38:52.645774 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.645733    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/761f0730-7b45-4b34-92c9-460d84a7f2be-root\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.645774 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.645785    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-accelerators-collector-config\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.646009 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.645815    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.646009 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.645825    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/761f0730-7b45-4b34-92c9-460d84a7f2be-root\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.646009 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.645843    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-wtmp\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.646009 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.645886    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-tls\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.646009 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.645931    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-textfile\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.646009 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.645955    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-wtmp\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.646009 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.645976    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/761f0730-7b45-4b34-92c9-460d84a7f2be-metrics-client-ca\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.646396 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.646013    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndqh\" (UniqueName: \"kubernetes.io/projected/761f0730-7b45-4b34-92c9-460d84a7f2be-kube-api-access-fndqh\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.646396 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.646050    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/761f0730-7b45-4b34-92c9-460d84a7f2be-sys\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.646396 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.646139    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/761f0730-7b45-4b34-92c9-460d84a7f2be-sys\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.647074 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.646448    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-textfile\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.647074 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.646823    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/761f0730-7b45-4b34-92c9-460d84a7f2be-metrics-client-ca\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.647074 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.646935    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-accelerators-collector-config\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.648398 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.648356    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.648750 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.648710    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/761f0730-7b45-4b34-92c9-460d84a7f2be-node-exporter-tls\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.703164 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.703133    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndqh\" (UniqueName: \"kubernetes.io/projected/761f0730-7b45-4b34-92c9-460d84a7f2be-kube-api-access-fndqh\") pod \"node-exporter-s2xhg\" (UID: \"761f0730-7b45-4b34-92c9-460d84a7f2be\") " pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.786991 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:52.786919    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s2xhg"
Oct 30 18:38:52.797985 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:38:52.797954    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761f0730_7b45_4b34_92c9_460d84a7f2be.slice/crio-4e7a6489dc94a2b20b04c749300d7c2072fe8197116884ce79e39fa26ed513de WatchSource:0}: Error finding container 4e7a6489dc94a2b20b04c749300d7c2072fe8197116884ce79e39fa26ed513de: Status 404 returned error can't find the container with id 4e7a6489dc94a2b20b04c749300d7c2072fe8197116884ce79e39fa26ed513de
Oct 30 18:38:53.368956 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:53.368920    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2xhg" event={"ID":"761f0730-7b45-4b34-92c9-460d84a7f2be","Type":"ContainerStarted","Data":"4e7a6489dc94a2b20b04c749300d7c2072fe8197116884ce79e39fa26ed513de"}
Oct 30 18:38:54.374387 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:54.374358    4267 generic.go:356] "Generic (PLEG): container finished" podID="761f0730-7b45-4b34-92c9-460d84a7f2be" containerID="42f633fe42cd64e6ae8ed2dffc2550b4af87cdbb9722028f87bec9d86b64495a" exitCode=0
Oct 30 18:38:54.374845 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:54.374398    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2xhg" event={"ID":"761f0730-7b45-4b34-92c9-460d84a7f2be","Type":"ContainerDied","Data":"42f633fe42cd64e6ae8ed2dffc2550b4af87cdbb9722028f87bec9d86b64495a"}
Oct 30 18:38:55.333341 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:55.333310    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zk6cm"
Oct 30 18:38:55.381026 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:55.380987    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2xhg" event={"ID":"761f0730-7b45-4b34-92c9-460d84a7f2be","Type":"ContainerStarted","Data":"ef86a3ae41b2f4b83255c2908acd921ce61bd68af56bd8e1ee716ac8bf49ff16"}
Oct 30 18:38:55.381468 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:55.381103    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s2xhg" event={"ID":"761f0730-7b45-4b34-92c9-460d84a7f2be","Type":"ContainerStarted","Data":"766c54546002e005425f2f36e93edf6accd73723d2a2cb3aba882e6d80e5347d"}
Oct 30 18:38:55.562340 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:55.562279    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s2xhg" podStartSLOduration=2.066191554 podStartE2EDuration="3.562260678s" totalImagesPullingTime="1.422749955s" totalInitContainerRuntime="73.319169ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:55.561911027 +0000 UTC m=+26.449828117" watchObservedRunningTime="2026-10-30 18:38:55.562260678 +0000 UTC m=+26.450177768"
Oct 30 18:38:58.348935 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:38:58.348901    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q4l7t"
Oct 30 18:39:00.021094 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:00.021065    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c75bw"
Oct 30 18:39:00.400280 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:00.400242    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-b8fk2" event={"ID":"986fe640-11d4-4f32-bba7-2c79ccd6067e","Type":"ContainerStarted","Data":"ee73c0aad1bd63b865ad31092045076bc06aff65033c524770c4074b6b3d5ec7"}
Oct 30 18:39:00.400511 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:00.400450    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-b8fk2"
Oct 30 18:39:00.420602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:00.420577    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-b8fk2"
Oct 30 18:39:00.514093 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:00.514009    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-b8fk2" podStartSLOduration=2.244348693 podStartE2EDuration="20.513994743s" totalImagesPullingTime="18.26964605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:00.513668542 +0000 UTC m=+31.401585632" watchObservedRunningTime="2026-10-30 18:39:00.513994743 +0000 UTC m=+31.401911831"
Oct 30 18:39:01.897485 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:01.897452    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:39:01.901925 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:01.901898    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:39:03.303230 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:03.303197    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c68fcb657-cq76w"
Oct 30 18:39:04.698002 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.697965    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67fb5fdbc9-4ntqp"]
Oct 30 18:39:04.728544 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.728487    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.778634 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.778596    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67fb5fdbc9-4ntqp"]
Oct 30 18:39:04.849637 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.849600    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-service-ca\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.849637 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.849649    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-config\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.849888 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.849675    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-oauth-config\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.849888 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.849836    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-oauth-serving-cert\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.849987 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.849884    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-tmp\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.849987 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.849917    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8zl4\" (UniqueName: \"kubernetes.io/projected/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-kube-api-access-s8zl4\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.850099 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.850033    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-serving-cert\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.850099 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.850064    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-trusted-ca-bundle\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.930559 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.930524    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 18:39:04.951056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.950962    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-oauth-config\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.951056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.951012    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-oauth-serving-cert\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.951056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.951050    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-tmp\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.951341 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.951080    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8zl4\" (UniqueName: \"kubernetes.io/projected/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-kube-api-access-s8zl4\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.951341 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.951121    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-serving-cert\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.951341 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.951145    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-trusted-ca-bundle\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.951341 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.951171    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-service-ca\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.951341 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.951205    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-config\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.952028 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.952000    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-config\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.952151 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.952124    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-tmp\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.952223 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.952195    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-oauth-serving-cert\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.952970 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.952840    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-service-ca\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.953079 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.953016    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-trusted-ca-bundle\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.968616 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.968426    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-oauth-config\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:04.969697 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:04.969671    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-serving-cert\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:05.008841 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:05.008798    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8zl4\" (UniqueName: \"kubernetes.io/projected/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-kube-api-access-s8zl4\") pod \"console-67fb5fdbc9-4ntqp\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") " pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:05.053426 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:05.053383    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:05.466658 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:05.466616    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67fb5fdbc9-4ntqp"]
Oct 30 18:39:05.499864 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:39:05.499834    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39608f2_10c3_4a43_a6b9_e0aaeb1f6e17.slice/crio-1d539720a6635dee84f483bb99bd7b9e0bf54c56532ec22c6362f5ac3bfc16d8 WatchSource:0}: Error finding container 1d539720a6635dee84f483bb99bd7b9e0bf54c56532ec22c6362f5ac3bfc16d8: Status 404 returned error can't find the container with id 1d539720a6635dee84f483bb99bd7b9e0bf54c56532ec22c6362f5ac3bfc16d8
Oct 30 18:39:06.425290 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:06.425250    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67fb5fdbc9-4ntqp" event={"ID":"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17","Type":"ContainerStarted","Data":"adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed"}
Oct 30 18:39:06.425860 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:06.425304    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67fb5fdbc9-4ntqp" event={"ID":"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17","Type":"ContainerStarted","Data":"1d539720a6635dee84f483bb99bd7b9e0bf54c56532ec22c6362f5ac3bfc16d8"}
Oct 30 18:39:15.054089 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:15.053972    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:15.054089 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:15.054055    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:15.058670 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:15.058648    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:15.173240 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:15.173184    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67fb5fdbc9-4ntqp" podStartSLOduration=11.173169561 podStartE2EDuration="11.173169561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:06.543252048 +0000 UTC m=+37.431169136" watchObservedRunningTime="2026-10-30 18:39:15.173169561 +0000 UTC m=+46.061086647"
Oct 30 18:39:15.459061 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:15.459033    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:39:15.624629 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:15.624593    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6599b687f8-sp2zz"]
Oct 30 18:39:20.156130 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:39:20.156090    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:39:40.644783 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:40.644724    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6599b687f8-sp2zz" podUID="a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" containerName="console" containerID="cri-o://1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be" gracePeriod=15
Oct 30 18:39:40.915514 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:40.915433    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:39:41.026778 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.026733    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-oauth-config\") pod \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") "
Oct 30 18:39:41.026935 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.026795    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-oauth-serving-cert\") pod \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") "
Oct 30 18:39:41.026935 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.026847    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-service-ca\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-service-ca\") pod \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") "
Oct 30 18:39:41.026935 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.026894    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-kube-api-access-hmj7n\" (UniqueName: \"kubernetes.io/projected/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-kube-api-access-hmj7n\") pod \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") "
Oct 30 18:39:41.026935 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.026914    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-tmp\") pod \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") "
Oct 30 18:39:41.027142 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.026946    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-serving-cert\") pod \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") "
Oct 30 18:39:41.027142 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.026981    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-config\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-config\") pod \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\" (UID: \"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d\") "
Oct 30 18:39:41.027282 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.027252    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-oauth-serving-cert" pod "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" (UID: "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:41.027587 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.027561    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-service-ca" pod "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" (UID: "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:41.027710 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.027261    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-tmp" pod "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" (UID: "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:41.027710 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.027474    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-config" pod "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" (UID: "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:41.029192 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.029163    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-serving-cert" pod "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" (UID: "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:41.029282 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.029226    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-kube-api-access-hmj7n" pod "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" (UID: "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d"). InnerVolumeSpecName "kube-api-access-hmj7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:41.029282 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.029253    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-oauth-config" pod "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" (UID: "a10d3e5b-debe-4c43-8eb5-c4be7e89a05d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:41.127869 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.127835    4267 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-oauth-config\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:41.127869 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.127864    4267 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-oauth-serving-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:41.127869 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.127875    4267 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:41.128107 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.127883    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmj7n\" (UniqueName: \"kubernetes.io/projected/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-kube-api-access-hmj7n\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:41.128107 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.127895    4267 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-tmp\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:41.128107 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.127903    4267 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-serving-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:41.128107 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.127911    4267 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d-console-config\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:39:41.542119 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.542022    4267 generic.go:356] "Generic (PLEG): container finished" podID="a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" containerID="1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be" exitCode=0
Oct 30 18:39:41.542119 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.542089    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6599b687f8-sp2zz"
Oct 30 18:39:41.542361 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.542092    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6599b687f8-sp2zz" event={"ID":"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d","Type":"ContainerDied","Data":"1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be"}
Oct 30 18:39:41.542361 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.542238    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6599b687f8-sp2zz" event={"ID":"a10d3e5b-debe-4c43-8eb5-c4be7e89a05d","Type":"ContainerDied","Data":"12765229e79153ba72a3977f1b8d30341a9e65fc5d1ab992f5e7bdb4358633cf"}
Oct 30 18:39:41.542361 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.542277    4267 scope.go:122] "RemoveContainer" containerID="1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be"
Oct 30 18:39:41.550461 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.550439    4267 scope.go:122] "RemoveContainer" containerID="1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be"
Oct 30 18:39:41.550760 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:39:41.550737    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be\": container with ID starting with 1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be not found: ID does not exist" containerID="1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be"
Oct 30 18:39:41.550846 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.550774    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be"} err="failed to get container status \"1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be\": rpc error: code = NotFound desc = could not find container \"1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be\": container with ID starting with 1344c9f407e3aa4082e1e741007f21cfbd7418c2673df192778be00f39ecf7be not found: ID does not exist"
Oct 30 18:39:41.703847 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.703812    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6599b687f8-sp2zz"]
Oct 30 18:39:41.707602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:41.707580    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6599b687f8-sp2zz"]
Oct 30 18:39:43.217249 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:43.217219    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" path="/var/lib/kubelet/pods/a10d3e5b-debe-4c43-8eb5-c4be7e89a05d/volumes"
Oct 30 18:39:48.292854 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.292822    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b5f8b84f4-zw4fx"]
Oct 30 18:39:48.293355 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.293067    4267 cpu_manager.go:414] "containerMap: removing container" podUID="a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" containerName="console"
Oct 30 18:39:48.293355 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.293078    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" containerName="console"
Oct 30 18:39:48.293355 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.293120    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10d3e5b-debe-4c43-8eb5-c4be7e89a05d" containerName="console"
Oct 30 18:39:48.295457 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.295440    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.372949 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.372914    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-oauth-serving-cert\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.372949 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.372956    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv62l\" (UniqueName: \"kubernetes.io/projected/e6e03037-0585-4e66-b6b7-38d3b359adec-kube-api-access-cv62l\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.373170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.372982    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-trusted-ca-bundle\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.373170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.373054    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6e03037-0585-4e66-b6b7-38d3b359adec-console-oauth-config\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.373170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.373095    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-console-config\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.373170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.373116    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6e03037-0585-4e66-b6b7-38d3b359adec-console-serving-cert\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.373170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.373135    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6e03037-0585-4e66-b6b7-38d3b359adec-tmp\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.373340 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.373201    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-service-ca\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.398303 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.398271    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5f8b84f4-zw4fx"]
Oct 30 18:39:48.474174 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.474137    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-service-ca\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.474367 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.474180    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-oauth-serving-cert\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.474367 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.474216    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv62l\" (UniqueName: \"kubernetes.io/projected/e6e03037-0585-4e66-b6b7-38d3b359adec-kube-api-access-cv62l\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.474367 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.474239    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-trusted-ca-bundle\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.474367 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.474272    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6e03037-0585-4e66-b6b7-38d3b359adec-console-oauth-config\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.474367 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.474319    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-console-config\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.474367 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.474348    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6e03037-0585-4e66-b6b7-38d3b359adec-console-serving-cert\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.474367 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.474368    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6e03037-0585-4e66-b6b7-38d3b359adec-tmp\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.474903 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.474868    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6e03037-0585-4e66-b6b7-38d3b359adec-tmp\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.475079 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.475055    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-oauth-serving-cert\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.475171 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.475055    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-service-ca\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.475231 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.475202    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-console-config\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.475347 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.475330    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e03037-0585-4e66-b6b7-38d3b359adec-trusted-ca-bundle\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.476910 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.476871    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6e03037-0585-4e66-b6b7-38d3b359adec-console-serving-cert\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.477023 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.476953    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6e03037-0585-4e66-b6b7-38d3b359adec-console-oauth-config\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.531552 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.531512    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv62l\" (UniqueName: \"kubernetes.io/projected/e6e03037-0585-4e66-b6b7-38d3b359adec-kube-api-access-cv62l\") pod \"console-6b5f8b84f4-zw4fx\" (UID: \"e6e03037-0585-4e66-b6b7-38d3b359adec\") " pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.605902 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.605817    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:48.971280 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:48.971246    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5f8b84f4-zw4fx"]
Oct 30 18:39:48.974368 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:39:48.974344    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e03037_0585_4e66_b6b7_38d3b359adec.slice/crio-d970ce8383b0a18ced2804ec2e698d4e32e47df9433a1581fd7609fe9a8f6cf6 WatchSource:0}: Error finding container d970ce8383b0a18ced2804ec2e698d4e32e47df9433a1581fd7609fe9a8f6cf6: Status 404 returned error can't find the container with id d970ce8383b0a18ced2804ec2e698d4e32e47df9433a1581fd7609fe9a8f6cf6
Oct 30 18:39:49.570504 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:49.570471    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5f8b84f4-zw4fx" event={"ID":"e6e03037-0585-4e66-b6b7-38d3b359adec","Type":"ContainerStarted","Data":"ce11ff2c0f371e4d79e4067fe7c8ca3e33cdde2152f43fdfd7b9cf911224ef43"}
Oct 30 18:39:49.570901 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:49.570551    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5f8b84f4-zw4fx" event={"ID":"e6e03037-0585-4e66-b6b7-38d3b359adec","Type":"ContainerStarted","Data":"d970ce8383b0a18ced2804ec2e698d4e32e47df9433a1581fd7609fe9a8f6cf6"}
Oct 30 18:39:49.688210 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:49.688159    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b5f8b84f4-zw4fx" podStartSLOduration=1.688146551 podStartE2EDuration="1.688146551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:49.687877408 +0000 UTC m=+80.575794499" watchObservedRunningTime="2026-10-30 18:39:49.688146551 +0000 UTC m=+80.576063639"
Oct 30 18:39:50.156094 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:39:50.156050    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:39:58.606287 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:58.606250    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:58.606795 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:58.606342    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:58.610944 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:58.610920    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:59.610510 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:59.610469    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b5f8b84f4-zw4fx"
Oct 30 18:39:59.774622 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:39:59.774594    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67fb5fdbc9-4ntqp"]
Oct 30 18:40:20.156853 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:40:20.156813    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:40:24.793753 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:24.793695    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67fb5fdbc9-4ntqp" podUID="d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" containerName="console" containerID="cri-o://adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed" gracePeriod=15
Oct 30 18:40:25.032902 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.032876    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:40:25.136589 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.136509    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-service-ca\") pod \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") "
Oct 30 18:40:25.136589 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.136544    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-config\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-config\") pod \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") "
Oct 30 18:40:25.136589 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.136589    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-serving-cert\") pod \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") "
Oct 30 18:40:25.136878 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.136629    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-oauth-serving-cert\") pod \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") "
Oct 30 18:40:25.136878 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.136648    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-tmp\") pod \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") "
Oct 30 18:40:25.136878 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.136676    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-oauth-config\") pod \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") "
Oct 30 18:40:25.136878 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.136713    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-kube-api-access-s8zl4\" (UniqueName: \"kubernetes.io/projected/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-kube-api-access-s8zl4\") pod \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") "
Oct 30 18:40:25.136878 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.136741    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-trusted-ca-bundle\") pod \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\" (UID: \"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17\") "
Oct 30 18:40:25.137125 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.136938    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-service-ca" pod "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" (UID: "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:40:25.137125 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.137099    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-tmp" pod "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" (UID: "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:40:25.137125 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.137111    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-config" pod "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" (UID: "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:40:25.137250 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.137144    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-oauth-serving-cert" pod "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" (UID: "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:40:25.137342 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.137320    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-trusted-ca-bundle" pod "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" (UID: "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:40:25.138840 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.138813    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-serving-cert" pod "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" (UID: "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:40:25.138957 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.138917    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-oauth-config" pod "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" (UID: "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:40:25.139000 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.138959    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-kube-api-access-s8zl4" pod "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" (UID: "d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17"). InnerVolumeSpecName "kube-api-access-s8zl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:40:25.238134 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.238105    4267 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-serving-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:40:25.238134 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.238130    4267 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-oauth-serving-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:40:25.238134 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.238140    4267 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-tmp\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:40:25.238374 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.238149    4267 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-oauth-config\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:40:25.238374 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.238158    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8zl4\" (UniqueName: \"kubernetes.io/projected/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-kube-api-access-s8zl4\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:40:25.238374 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.238169    4267 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-trusted-ca-bundle\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:40:25.238374 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.238177    4267 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:40:25.238374 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.238187    4267 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17-console-config\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:40:25.687862 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.687825    4267 generic.go:356] "Generic (PLEG): container finished" podID="d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" containerID="adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed" exitCode=0
Oct 30 18:40:25.688041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.687869    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67fb5fdbc9-4ntqp" event={"ID":"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17","Type":"ContainerDied","Data":"adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed"}
Oct 30 18:40:25.688041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.687875    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67fb5fdbc9-4ntqp"
Oct 30 18:40:25.688041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.687911    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67fb5fdbc9-4ntqp" event={"ID":"d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17","Type":"ContainerDied","Data":"1d539720a6635dee84f483bb99bd7b9e0bf54c56532ec22c6362f5ac3bfc16d8"}
Oct 30 18:40:25.688041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.687930    4267 scope.go:122] "RemoveContainer" containerID="adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed"
Oct 30 18:40:25.695653 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.695635    4267 scope.go:122] "RemoveContainer" containerID="adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed"
Oct 30 18:40:25.695935 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:40:25.695916    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed\": container with ID starting with adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed not found: ID does not exist" containerID="adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed"
Oct 30 18:40:25.695976 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.695943    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed"} err="failed to get container status \"adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed\": rpc error: code = NotFound desc = could not find container \"adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed\": container with ID starting with adea677d4c5c524a4958729dd0a671e7818d5410ba608c18755e2736c506deed not found: ID does not exist"
Oct 30 18:40:25.849700 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.849671    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67fb5fdbc9-4ntqp"]
Oct 30 18:40:25.899270 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:25.899238    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67fb5fdbc9-4ntqp"]
Oct 30 18:40:27.217423 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:40:27.217386    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" path="/var/lib/kubelet/pods/d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17/volumes"
Oct 30 18:40:50.157059 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:40:50.156973    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:41:20.156956 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:41:20.156915    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:41:50.156264 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:41:50.156224    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:42:10.349135 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.349089    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"]
Oct 30 18:42:10.349766 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.349366    4267 cpu_manager.go:414] "containerMap: removing container" podUID="d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" containerName="console"
Oct 30 18:42:10.349766 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.349379    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" containerName="console"
Oct 30 18:42:10.349766 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.349457    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39608f2-10c3-4a43-a6b9-e0aaeb1f6e17" containerName="console"
Oct 30 18:42:10.351204 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.351187    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:10.452903 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.452867    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:42:10.452903 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.452868    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:42:10.453197 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.453106    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-wjzfc\""
Oct 30 18:42:10.454045 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.454026    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"]
Oct 30 18:42:10.471040 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.471009    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:10.471180 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.471051    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shnvh\" (UniqueName: \"kubernetes.io/projected/12bb682d-d35e-4463-adc0-d4c672a1f8a5-kube-api-access-shnvh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:10.471180 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.471073    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:10.571784 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.571749    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:10.571999 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.571795    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shnvh\" (UniqueName: \"kubernetes.io/projected/12bb682d-d35e-4463-adc0-d4c672a1f8a5-kube-api-access-shnvh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:10.571999 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.571907    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:10.572142 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.572125    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:10.572267 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.572246    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:10.629035 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.628973    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shnvh\" (UniqueName: \"kubernetes.io/projected/12bb682d-d35e-4463-adc0-d4c672a1f8a5-kube-api-access-shnvh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:10.663087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:10.663050    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:11.028842 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:11.028770    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"]
Oct 30 18:42:11.031799 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:42:11.031763    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bb682d_d35e_4463_adc0_d4c672a1f8a5.slice/crio-fdff62a5500bea554070b9f3a23aef25673cf4baad234c8d758780646e1581db WatchSource:0}: Error finding container fdff62a5500bea554070b9f3a23aef25673cf4baad234c8d758780646e1581db: Status 404 returned error can't find the container with id fdff62a5500bea554070b9f3a23aef25673cf4baad234c8d758780646e1581db
Oct 30 18:42:12.014744 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:12.014704    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b" event={"ID":"12bb682d-d35e-4463-adc0-d4c672a1f8a5","Type":"ContainerStarted","Data":"fdff62a5500bea554070b9f3a23aef25673cf4baad234c8d758780646e1581db"}
Oct 30 18:42:19.040465 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:19.040422    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b" event={"ID":"12bb682d-d35e-4463-adc0-d4c672a1f8a5","Type":"ContainerStarted","Data":"f66e8c2b36f54772bad06b97f3a95f6118f767f584b7f28496bb6805c84dbf9f"}
Oct 30 18:42:20.045302 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:20.045267    4267 generic.go:356] "Generic (PLEG): container finished" podID="12bb682d-d35e-4463-adc0-d4c672a1f8a5" containerID="f66e8c2b36f54772bad06b97f3a95f6118f767f584b7f28496bb6805c84dbf9f" exitCode=0
Oct 30 18:42:20.045735 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:20.045359    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b" event={"ID":"12bb682d-d35e-4463-adc0-d4c672a1f8a5","Type":"ContainerDied","Data":"f66e8c2b36f54772bad06b97f3a95f6118f767f584b7f28496bb6805c84dbf9f"}
Oct 30 18:42:20.156876 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:42:20.156839    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:42:24.060088 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:24.060049    4267 generic.go:356] "Generic (PLEG): container finished" podID="12bb682d-d35e-4463-adc0-d4c672a1f8a5" containerID="78f3509ef03cbdb823de6a11e9f021d84928e35e2f6f0259ffa609d58b79970d" exitCode=0
Oct 30 18:42:24.060509 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:24.060141    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b" event={"ID":"12bb682d-d35e-4463-adc0-d4c672a1f8a5","Type":"ContainerDied","Data":"78f3509ef03cbdb823de6a11e9f021d84928e35e2f6f0259ffa609d58b79970d"}
Oct 30 18:42:31.087296 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:31.087255    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b" event={"ID":"12bb682d-d35e-4463-adc0-d4c672a1f8a5","Type":"ContainerStarted","Data":"614eab4ec8663267bdf52c404a764e4d5e7652799674b0989b7b01a6d3974150"}
Oct 30 18:42:31.203201 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:31.203142    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b" podStartSLOduration=3.7586617589999998 podStartE2EDuration="21.203125093s" totalImagesPullingTime="17.444463334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:31.20258468 +0000 UTC m=+242.090501769" watchObservedRunningTime="2026-10-30 18:42:31.203125093 +0000 UTC m=+242.091042183"
Oct 30 18:42:32.093199 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:32.093160    4267 generic.go:356] "Generic (PLEG): container finished" podID="12bb682d-d35e-4463-adc0-d4c672a1f8a5" containerID="614eab4ec8663267bdf52c404a764e4d5e7652799674b0989b7b01a6d3974150" exitCode=0
Oct 30 18:42:32.093640 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:32.093240    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b" event={"ID":"12bb682d-d35e-4463-adc0-d4c672a1f8a5","Type":"ContainerDied","Data":"614eab4ec8663267bdf52c404a764e4d5e7652799674b0989b7b01a6d3974150"}
Oct 30 18:42:33.260549 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:33.260520    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:33.346005 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:33.345969    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-bundle\") pod \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") "
Oct 30 18:42:33.346005 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:33.346009    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-util\" (UniqueName: \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-util\") pod \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") "
Oct 30 18:42:33.346258 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:33.346040    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bb682d-d35e-4463-adc0-d4c672a1f8a5-kube-api-access-shnvh\" (UniqueName: \"kubernetes.io/projected/12bb682d-d35e-4463-adc0-d4c672a1f8a5-kube-api-access-shnvh\") pod \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\" (UID: \"12bb682d-d35e-4463-adc0-d4c672a1f8a5\") "
Oct 30 18:42:33.346843 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:33.346726    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-bundle" pod "12bb682d-d35e-4463-adc0-d4c672a1f8a5" (UID: "12bb682d-d35e-4463-adc0-d4c672a1f8a5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:33.348352 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:33.348331    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bb682d-d35e-4463-adc0-d4c672a1f8a5-kube-api-access-shnvh" pod "12bb682d-d35e-4463-adc0-d4c672a1f8a5" (UID: "12bb682d-d35e-4463-adc0-d4c672a1f8a5"). InnerVolumeSpecName "kube-api-access-shnvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:33.351392 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:33.351360    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-util" pod "12bb682d-d35e-4463-adc0-d4c672a1f8a5" (UID: "12bb682d-d35e-4463-adc0-d4c672a1f8a5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:33.447685 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:33.447644    4267 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-bundle\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:42:33.447685 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:33.447677    4267 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12bb682d-d35e-4463-adc0-d4c672a1f8a5-util\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:42:33.447685 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:33.447688    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shnvh\" (UniqueName: \"kubernetes.io/projected/12bb682d-d35e-4463-adc0-d4c672a1f8a5-kube-api-access-shnvh\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:42:34.101318 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:34.101283    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b" event={"ID":"12bb682d-d35e-4463-adc0-d4c672a1f8a5","Type":"ContainerDied","Data":"fdff62a5500bea554070b9f3a23aef25673cf4baad234c8d758780646e1581db"}
Oct 30 18:42:34.101318 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:34.101302    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwp26b"
Oct 30 18:42:34.101548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:34.101335    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdff62a5500bea554070b9f3a23aef25673cf4baad234c8d758780646e1581db"
Oct 30 18:42:38.308363 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.308326    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"]
Oct 30 18:42:38.308799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.308576    4267 cpu_manager.go:414] "containerMap: removing container" podUID="12bb682d-d35e-4463-adc0-d4c672a1f8a5" containerName="util"
Oct 30 18:42:38.308799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.308587    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bb682d-d35e-4463-adc0-d4c672a1f8a5" containerName="util"
Oct 30 18:42:38.308799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.308596    4267 cpu_manager.go:414] "containerMap: removing container" podUID="12bb682d-d35e-4463-adc0-d4c672a1f8a5" containerName="pull"
Oct 30 18:42:38.308799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.308602    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bb682d-d35e-4463-adc0-d4c672a1f8a5" containerName="pull"
Oct 30 18:42:38.308799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.308613    4267 cpu_manager.go:414] "containerMap: removing container" podUID="12bb682d-d35e-4463-adc0-d4c672a1f8a5" containerName="extract"
Oct 30 18:42:38.308799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.308618    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bb682d-d35e-4463-adc0-d4c672a1f8a5" containerName="extract"
Oct 30 18:42:38.308799 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.308664    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bb682d-d35e-4463-adc0-d4c672a1f8a5" containerName="extract"
Oct 30 18:42:38.310537 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.310520    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"
Oct 30 18:42:38.363508 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.363458    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"]
Oct 30 18:42:38.385689 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.385656    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2175e554-4c1b-44c3-805e-a63a2eda1f4f-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-b4hs9\" (UID: \"2175e554-4c1b-44c3-805e-a63a2eda1f4f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"
Oct 30 18:42:38.385868 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.385697    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdbb9\" (UniqueName: \"kubernetes.io/projected/2175e554-4c1b-44c3-805e-a63a2eda1f4f-kube-api-access-mdbb9\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-b4hs9\" (UID: \"2175e554-4c1b-44c3-805e-a63a2eda1f4f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"
Oct 30 18:42:38.412006 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.411971    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:42:38.412006 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.411984    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 18:42:38.412006 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.411998    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lxd5n\""
Oct 30 18:42:38.486556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.486520    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2175e554-4c1b-44c3-805e-a63a2eda1f4f-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-b4hs9\" (UID: \"2175e554-4c1b-44c3-805e-a63a2eda1f4f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"
Oct 30 18:42:38.486707 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.486563    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdbb9\" (UniqueName: \"kubernetes.io/projected/2175e554-4c1b-44c3-805e-a63a2eda1f4f-kube-api-access-mdbb9\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-b4hs9\" (UID: \"2175e554-4c1b-44c3-805e-a63a2eda1f4f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"
Oct 30 18:42:38.486899 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.486880    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2175e554-4c1b-44c3-805e-a63a2eda1f4f-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-b4hs9\" (UID: \"2175e554-4c1b-44c3-805e-a63a2eda1f4f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"
Oct 30 18:42:38.543135 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.543104    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdbb9\" (UniqueName: \"kubernetes.io/projected/2175e554-4c1b-44c3-805e-a63a2eda1f4f-kube-api-access-mdbb9\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-b4hs9\" (UID: \"2175e554-4c1b-44c3-805e-a63a2eda1f4f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"
Oct 30 18:42:38.620181 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:38.620106    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"
Oct 30 18:42:39.085998 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:39.085973    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9"]
Oct 30 18:42:39.088422 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:42:39.088393    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2175e554_4c1b_44c3_805e_a63a2eda1f4f.slice/crio-0b14c0025dd75878b9f081cee08b58b1fabf7f3e0f82b45fd17a8f0550894c17 WatchSource:0}: Error finding container 0b14c0025dd75878b9f081cee08b58b1fabf7f3e0f82b45fd17a8f0550894c17: Status 404 returned error can't find the container with id 0b14c0025dd75878b9f081cee08b58b1fabf7f3e0f82b45fd17a8f0550894c17
Oct 30 18:42:39.123141 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:39.123112    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9" event={"ID":"2175e554-4c1b-44c3-805e-a63a2eda1f4f","Type":"ContainerStarted","Data":"0b14c0025dd75878b9f081cee08b58b1fabf7f3e0f82b45fd17a8f0550894c17"}
Oct 30 18:42:42.142124 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:42.142040    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9" event={"ID":"2175e554-4c1b-44c3-805e-a63a2eda1f4f","Type":"ContainerStarted","Data":"175dc779a0a7e1faa032d0e1ab18cb65f3fb0ca29ba546a24a453670424062b8"}
Oct 30 18:42:42.355791 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:42.355737    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-b4hs9" podStartSLOduration=1.734690627 podStartE2EDuration="4.355723858s" totalImagesPullingTime="2.621033231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:42.305613092 +0000 UTC m=+253.193530181" watchObservedRunningTime="2026-10-30 18:42:42.355723858 +0000 UTC m=+253.243640940"
Oct 30 18:42:43.116889 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.116848    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"]
Oct 30 18:42:43.119124 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.119102    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.179961 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.179928    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"]
Oct 30 18:42:43.218470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.218439    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.218682 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.218515    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.218682 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.218632    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqfd4\" (UniqueName: \"kubernetes.io/projected/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-kube-api-access-tqfd4\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.222199 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.222172    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:42:43.222339 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.222220    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-h8kf9\""
Oct 30 18:42:43.222339 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.222255    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:42:43.319835 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.319787    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.320018 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.319854    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqfd4\" (UniqueName: \"kubernetes.io/projected/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-kube-api-access-tqfd4\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.320018 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.319908    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.320211 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.320191    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.320271 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.320251    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.380174 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.380100    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqfd4\" (UniqueName: \"kubernetes.io/projected/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-kube-api-access-tqfd4\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.431559 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.431526    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:43.810265 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:43.810232    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"]
Oct 30 18:42:43.836193 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:42:43.836163    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8fe4ce_c1ed_48ef_954a_4dcf842c7610.slice/crio-60c89ccf360c0ed619c5a0bc1cfa67aee4f36bd070da18186de2405df0e03e99 WatchSource:0}: Error finding container 60c89ccf360c0ed619c5a0bc1cfa67aee4f36bd070da18186de2405df0e03e99: Status 404 returned error can't find the container with id 60c89ccf360c0ed619c5a0bc1cfa67aee4f36bd070da18186de2405df0e03e99
Oct 30 18:42:44.106022 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.105946    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"]
Oct 30 18:42:44.107919 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.107904    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"
Oct 30 18:42:44.150610 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.150577    4267 generic.go:356] "Generic (PLEG): container finished" podID="ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" containerID="1a91664d7dad3c49e588e73a46ba287f80317069104904a2fc180745b43f7dcd" exitCode=0
Oct 30 18:42:44.150758 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.150639    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g" event={"ID":"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610","Type":"ContainerDied","Data":"1a91664d7dad3c49e588e73a46ba287f80317069104904a2fc180745b43f7dcd"}
Oct 30 18:42:44.150758 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.150676    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g" event={"ID":"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610","Type":"ContainerStarted","Data":"60c89ccf360c0ed619c5a0bc1cfa67aee4f36bd070da18186de2405df0e03e99"}
Oct 30 18:42:44.160347 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.160325    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 18:42:44.160459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.160383    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 18:42:44.160459 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.160389    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mjpmn\""
Oct 30 18:42:44.166451 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.166415    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"]
Oct 30 18:42:44.224609 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.224572    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acd523a4-65be-45f7-a79a-5c642c8f4787-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vnbwf\" (UID: \"acd523a4-65be-45f7-a79a-5c642c8f4787\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"
Oct 30 18:42:44.224997 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.224643    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv2sd\" (UniqueName: \"kubernetes.io/projected/acd523a4-65be-45f7-a79a-5c642c8f4787-kube-api-access-gv2sd\") pod \"cert-manager-webhook-5dbbbf75cc-vnbwf\" (UID: \"acd523a4-65be-45f7-a79a-5c642c8f4787\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"
Oct 30 18:42:44.325621 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.325591    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acd523a4-65be-45f7-a79a-5c642c8f4787-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vnbwf\" (UID: \"acd523a4-65be-45f7-a79a-5c642c8f4787\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"
Oct 30 18:42:44.325795 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.325637    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv2sd\" (UniqueName: \"kubernetes.io/projected/acd523a4-65be-45f7-a79a-5c642c8f4787-kube-api-access-gv2sd\") pod \"cert-manager-webhook-5dbbbf75cc-vnbwf\" (UID: \"acd523a4-65be-45f7-a79a-5c642c8f4787\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"
Oct 30 18:42:44.382308 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.382242    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acd523a4-65be-45f7-a79a-5c642c8f4787-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vnbwf\" (UID: \"acd523a4-65be-45f7-a79a-5c642c8f4787\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"
Oct 30 18:42:44.382432 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.382363    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv2sd\" (UniqueName: \"kubernetes.io/projected/acd523a4-65be-45f7-a79a-5c642c8f4787-kube-api-access-gv2sd\") pod \"cert-manager-webhook-5dbbbf75cc-vnbwf\" (UID: \"acd523a4-65be-45f7-a79a-5c642c8f4787\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"
Oct 30 18:42:44.430453 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.430413    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"
Oct 30 18:42:44.801225 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:44.801200    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"]
Oct 30 18:42:44.803721 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:42:44.803693    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacd523a4_65be_45f7_a79a_5c642c8f4787.slice/crio-3ef9dfacca57c748adc5a92332c94e631f267875ef0da8228aa4700c63695787 WatchSource:0}: Error finding container 3ef9dfacca57c748adc5a92332c94e631f267875ef0da8228aa4700c63695787: Status 404 returned error can't find the container with id 3ef9dfacca57c748adc5a92332c94e631f267875ef0da8228aa4700c63695787
Oct 30 18:42:45.155914 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:45.155834    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf" event={"ID":"acd523a4-65be-45f7-a79a-5c642c8f4787","Type":"ContainerStarted","Data":"3ef9dfacca57c748adc5a92332c94e631f267875ef0da8228aa4700c63695787"}
Oct 30 18:42:46.278383 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.278346    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"]
Oct 30 18:42:46.280605 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.280580    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"
Oct 30 18:42:46.333228 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.333181    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-t59cf\""
Oct 30 18:42:46.384244 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.384203    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"]
Oct 30 18:42:46.444652 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.444617    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45t9p\" (UniqueName: \"kubernetes.io/projected/0dd43aa8-8640-47d6-8e07-ff1f3be092a6-kube-api-access-45t9p\") pod \"cert-manager-cainjector-5c57bbd4fc-vm4d7\" (UID: \"0dd43aa8-8640-47d6-8e07-ff1f3be092a6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"
Oct 30 18:42:46.444839 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.444667    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dd43aa8-8640-47d6-8e07-ff1f3be092a6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vm4d7\" (UID: \"0dd43aa8-8640-47d6-8e07-ff1f3be092a6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"
Oct 30 18:42:46.545176 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.545096    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45t9p\" (UniqueName: \"kubernetes.io/projected/0dd43aa8-8640-47d6-8e07-ff1f3be092a6-kube-api-access-45t9p\") pod \"cert-manager-cainjector-5c57bbd4fc-vm4d7\" (UID: \"0dd43aa8-8640-47d6-8e07-ff1f3be092a6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"
Oct 30 18:42:46.545339 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.545178    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dd43aa8-8640-47d6-8e07-ff1f3be092a6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vm4d7\" (UID: \"0dd43aa8-8640-47d6-8e07-ff1f3be092a6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"
Oct 30 18:42:46.603460 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.603427    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45t9p\" (UniqueName: \"kubernetes.io/projected/0dd43aa8-8640-47d6-8e07-ff1f3be092a6-kube-api-access-45t9p\") pod \"cert-manager-cainjector-5c57bbd4fc-vm4d7\" (UID: \"0dd43aa8-8640-47d6-8e07-ff1f3be092a6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"
Oct 30 18:42:46.603661 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.603565    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dd43aa8-8640-47d6-8e07-ff1f3be092a6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vm4d7\" (UID: \"0dd43aa8-8640-47d6-8e07-ff1f3be092a6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"
Oct 30 18:42:46.893015 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:46.892932    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"
Oct 30 18:42:47.277542 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:47.277504    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7"]
Oct 30 18:42:47.281429 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:42:47.281396    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dd43aa8_8640_47d6_8e07_ff1f3be092a6.slice/crio-c530892e6aa164ac45f2244ff1b896a3e4efe8c2a5f32423a4019019beabcb72 WatchSource:0}: Error finding container c530892e6aa164ac45f2244ff1b896a3e4efe8c2a5f32423a4019019beabcb72: Status 404 returned error can't find the container with id c530892e6aa164ac45f2244ff1b896a3e4efe8c2a5f32423a4019019beabcb72
Oct 30 18:42:48.171022 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:48.170984    4267 generic.go:356] "Generic (PLEG): container finished" podID="ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" containerID="00944584cf8607ee64467658004ef81933a637bf312b3419fec2e5521771a592" exitCode=0
Oct 30 18:42:48.171308 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:48.171283    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g" event={"ID":"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610","Type":"ContainerDied","Data":"00944584cf8607ee64467658004ef81933a637bf312b3419fec2e5521771a592"}
Oct 30 18:42:48.172834 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:48.172787    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7" event={"ID":"0dd43aa8-8640-47d6-8e07-ff1f3be092a6","Type":"ContainerStarted","Data":"c530892e6aa164ac45f2244ff1b896a3e4efe8c2a5f32423a4019019beabcb72"}
Oct 30 18:42:49.179170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:49.179135    4267 generic.go:356] "Generic (PLEG): container finished" podID="ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" containerID="4e9d17b353f296edc86da60693a561fcee6e150cdafd224091ea76433d3393cb" exitCode=0
Oct 30 18:42:49.179670 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:49.179227    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g" event={"ID":"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610","Type":"ContainerDied","Data":"4e9d17b353f296edc86da60693a561fcee6e150cdafd224091ea76433d3393cb"}
Oct 30 18:42:49.180585 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:49.180562    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf" event={"ID":"acd523a4-65be-45f7-a79a-5c642c8f4787","Type":"ContainerStarted","Data":"d1b00a427b8bbabb942215a682d0328bcebcff34ed5147263abace34797ba35a"}
Oct 30 18:42:49.180680 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:49.180621    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"
Oct 30 18:42:49.181937 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:49.181916    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7" event={"ID":"0dd43aa8-8640-47d6-8e07-ff1f3be092a6","Type":"ContainerStarted","Data":"f9471b3aff361d1de66d996d8f9b6704051bb30f87431f8918fae3985779a627"}
Oct 30 18:42:49.410844 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:49.410777    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vm4d7" podStartSLOduration=2.084035303 podStartE2EDuration="3.410759381s" totalImagesPullingTime="1.326724078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:49.410059921 +0000 UTC m=+260.297977009" watchObservedRunningTime="2026-10-30 18:42:49.410759381 +0000 UTC m=+260.298676471"
Oct 30 18:42:49.526766 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:49.526699    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf" podStartSLOduration=1.968249825 podStartE2EDuration="5.526681412s" totalImagesPullingTime="3.558431587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:49.526379796 +0000 UTC m=+260.414296886" watchObservedRunningTime="2026-10-30 18:42:49.526681412 +0000 UTC m=+260.414598502"
Oct 30 18:42:50.156645 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:42:50.156604    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:42:50.309463 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:50.309438    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:50.477066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:50.477033    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-bundle\") pod \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") "
Oct 30 18:42:50.477265 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:50.477095    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-kube-api-access-tqfd4\" (UniqueName: \"kubernetes.io/projected/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-kube-api-access-tqfd4\") pod \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") "
Oct 30 18:42:50.477265 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:50.477120    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-util\" (UniqueName: \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-util\") pod \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\" (UID: \"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610\") "
Oct 30 18:42:50.477548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:50.477481    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-bundle" pod "ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" (UID: "ec8fe4ce-c1ed-48ef-954a-4dcf842c7610"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:50.479202 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:50.479181    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-kube-api-access-tqfd4" pod "ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" (UID: "ec8fe4ce-c1ed-48ef-954a-4dcf842c7610"). InnerVolumeSpecName "kube-api-access-tqfd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:50.481401 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:50.481382    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-util" pod "ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" (UID: "ec8fe4ce-c1ed-48ef-954a-4dcf842c7610"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:50.578289 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:50.578250    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqfd4\" (UniqueName: \"kubernetes.io/projected/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-kube-api-access-tqfd4\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:42:50.578289 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:50.578284    4267 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-util\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:42:50.578289 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:50.578293    4267 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec8fe4ce-c1ed-48ef-954a-4dcf842c7610-bundle\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:42:51.190854 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:51.190826    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g"
Oct 30 18:42:51.191025 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:51.190826    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc8558g6g" event={"ID":"ec8fe4ce-c1ed-48ef-954a-4dcf842c7610","Type":"ContainerDied","Data":"60c89ccf360c0ed619c5a0bc1cfa67aee4f36bd070da18186de2405df0e03e99"}
Oct 30 18:42:51.191025 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:51.190965    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60c89ccf360c0ed619c5a0bc1cfa67aee4f36bd070da18186de2405df0e03e99"
Oct 30 18:42:55.187805 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:42:55.187776    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf"
Oct 30 18:43:04.392384 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.392346    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"]
Oct 30 18:43:04.392876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.392627    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" containerName="util"
Oct 30 18:43:04.392876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.392640    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" containerName="util"
Oct 30 18:43:04.392876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.392654    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" containerName="extract"
Oct 30 18:43:04.392876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.392661    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" containerName="extract"
Oct 30 18:43:04.392876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.392669    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" containerName="pull"
Oct 30 18:43:04.392876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.392674    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" containerName="pull"
Oct 30 18:43:04.392876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.392723    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8fe4ce-c1ed-48ef-954a-4dcf842c7610" containerName="extract"
Oct 30 18:43:04.427662 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.427634    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:04.453380 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.453354    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"]
Oct 30 18:43:04.501469 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.501431    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:43:04.501668 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.501543    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:43:04.501668 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.501563    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-fkhmx\""
Oct 30 18:43:04.578584 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.578544    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9r9\" (UniqueName: \"kubernetes.io/projected/397230eb-ecbf-4db6-8183-30d8b5d9b59e-kube-api-access-2w9r9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:04.578751 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.578594    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:04.578751 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.578685    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:04.679241 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.679159    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9r9\" (UniqueName: \"kubernetes.io/projected/397230eb-ecbf-4db6-8183-30d8b5d9b59e-kube-api-access-2w9r9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:04.679241 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.679206    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:04.679241 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.679229    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:04.679636 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.679609    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:04.679782 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.679707    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:04.735639 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.735611    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9r9\" (UniqueName: \"kubernetes.io/projected/397230eb-ecbf-4db6-8183-30d8b5d9b59e-kube-api-access-2w9r9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:04.778458 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:04.778426    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:05.143843 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:05.143701    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"]
Oct 30 18:43:05.146311 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:43:05.146282    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397230eb_ecbf_4db6_8183_30d8b5d9b59e.slice/crio-2e333b9fb3a7e5b6b228e852c25a8d5ff81d213941b9d037f00936dd66aa40c0 WatchSource:0}: Error finding container 2e333b9fb3a7e5b6b228e852c25a8d5ff81d213941b9d037f00936dd66aa40c0: Status 404 returned error can't find the container with id 2e333b9fb3a7e5b6b228e852c25a8d5ff81d213941b9d037f00936dd66aa40c0
Oct 30 18:43:05.241390 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:05.241361    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss" event={"ID":"397230eb-ecbf-4db6-8183-30d8b5d9b59e","Type":"ContainerStarted","Data":"cef84f26aa6c17a174180372ca94b838761a56ca202e8d0880e50b8e842618a9"}
Oct 30 18:43:05.241527 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:05.241418    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss" event={"ID":"397230eb-ecbf-4db6-8183-30d8b5d9b59e","Type":"ContainerStarted","Data":"2e333b9fb3a7e5b6b228e852c25a8d5ff81d213941b9d037f00936dd66aa40c0"}
Oct 30 18:43:06.246711 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:06.246678    4267 generic.go:356] "Generic (PLEG): container finished" podID="397230eb-ecbf-4db6-8183-30d8b5d9b59e" containerID="cef84f26aa6c17a174180372ca94b838761a56ca202e8d0880e50b8e842618a9" exitCode=0
Oct 30 18:43:06.246711 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:06.246703    4267 generic.go:356] "Generic (PLEG): container finished" podID="397230eb-ecbf-4db6-8183-30d8b5d9b59e" containerID="1ce9ce2bcef97c1a9c5c0f9c4f33549875102df28b3a712f2ce6d32a2113a4a8" exitCode=0
Oct 30 18:43:06.247234 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:06.246760    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss" event={"ID":"397230eb-ecbf-4db6-8183-30d8b5d9b59e","Type":"ContainerDied","Data":"cef84f26aa6c17a174180372ca94b838761a56ca202e8d0880e50b8e842618a9"}
Oct 30 18:43:06.247234 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:06.246823    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss" event={"ID":"397230eb-ecbf-4db6-8183-30d8b5d9b59e","Type":"ContainerDied","Data":"1ce9ce2bcef97c1a9c5c0f9c4f33549875102df28b3a712f2ce6d32a2113a4a8"}
Oct 30 18:43:07.253758 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:07.253724    4267 generic.go:356] "Generic (PLEG): container finished" podID="397230eb-ecbf-4db6-8183-30d8b5d9b59e" containerID="fea475ffea96bf52e4bc25c5b7a7a5b607255574b189facda4302347916eb61b" exitCode=0
Oct 30 18:43:07.254139 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:07.253807    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss" event={"ID":"397230eb-ecbf-4db6-8183-30d8b5d9b59e","Type":"ContainerDied","Data":"fea475ffea96bf52e4bc25c5b7a7a5b607255574b189facda4302347916eb61b"}
Oct 30 18:43:08.379528 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:08.379483    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:08.508288 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:08.508253    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397230eb-ecbf-4db6-8183-30d8b5d9b59e-kube-api-access-2w9r9\" (UniqueName: \"kubernetes.io/projected/397230eb-ecbf-4db6-8183-30d8b5d9b59e-kube-api-access-2w9r9\") pod \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") "
Oct 30 18:43:08.508454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:08.508328    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-bundle\") pod \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") "
Oct 30 18:43:08.508454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:08.508359    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-util\" (UniqueName: \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-util\") pod \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\" (UID: \"397230eb-ecbf-4db6-8183-30d8b5d9b59e\") "
Oct 30 18:43:08.508903 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:08.508871    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-bundle" pod "397230eb-ecbf-4db6-8183-30d8b5d9b59e" (UID: "397230eb-ecbf-4db6-8183-30d8b5d9b59e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:08.510300 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:08.510273    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397230eb-ecbf-4db6-8183-30d8b5d9b59e-kube-api-access-2w9r9" pod "397230eb-ecbf-4db6-8183-30d8b5d9b59e" (UID: "397230eb-ecbf-4db6-8183-30d8b5d9b59e"). InnerVolumeSpecName "kube-api-access-2w9r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:08.514294 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:08.514271    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-util" pod "397230eb-ecbf-4db6-8183-30d8b5d9b59e" (UID: "397230eb-ecbf-4db6-8183-30d8b5d9b59e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:08.609155 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:08.609127    4267 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-bundle\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:43:08.609155 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:08.609152    4267 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/397230eb-ecbf-4db6-8183-30d8b5d9b59e-util\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:43:08.609287 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:08.609167    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w9r9\" (UniqueName: \"kubernetes.io/projected/397230eb-ecbf-4db6-8183-30d8b5d9b59e-kube-api-access-2w9r9\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:43:09.263718 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:09.263690    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss"
Oct 30 18:43:09.263718 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:09.263697    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpnss" event={"ID":"397230eb-ecbf-4db6-8183-30d8b5d9b59e","Type":"ContainerDied","Data":"2e333b9fb3a7e5b6b228e852c25a8d5ff81d213941b9d037f00936dd66aa40c0"}
Oct 30 18:43:09.263921 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:09.263760    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e333b9fb3a7e5b6b228e852c25a8d5ff81d213941b9d037f00936dd66aa40c0"
Oct 30 18:43:16.786380 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.786345    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"]
Oct 30 18:43:16.786819 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.786686    4267 cpu_manager.go:414] "containerMap: removing container" podUID="397230eb-ecbf-4db6-8183-30d8b5d9b59e" containerName="pull"
Oct 30 18:43:16.786819 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.786704    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397230eb-ecbf-4db6-8183-30d8b5d9b59e" containerName="pull"
Oct 30 18:43:16.786819 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.786714    4267 cpu_manager.go:414] "containerMap: removing container" podUID="397230eb-ecbf-4db6-8183-30d8b5d9b59e" containerName="extract"
Oct 30 18:43:16.786819 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.786719    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397230eb-ecbf-4db6-8183-30d8b5d9b59e" containerName="extract"
Oct 30 18:43:16.786819 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.786730    4267 cpu_manager.go:414] "containerMap: removing container" podUID="397230eb-ecbf-4db6-8183-30d8b5d9b59e" containerName="util"
Oct 30 18:43:16.786819 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.786736    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397230eb-ecbf-4db6-8183-30d8b5d9b59e" containerName="util"
Oct 30 18:43:16.786819 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.786782    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="397230eb-ecbf-4db6-8183-30d8b5d9b59e" containerName="extract"
Oct 30 18:43:16.788701 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.788684    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.864307 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.864263    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddkf8\" (UniqueName: \"kubernetes.io/projected/276385b5-61ba-4c52-a013-ba32fdccafdc-kube-api-access-ddkf8\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.864307 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.864308    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/276385b5-61ba-4c52-a013-ba32fdccafdc-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.864570 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.864325    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/276385b5-61ba-4c52-a013-ba32fdccafdc-manifests\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.864570 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.864380    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/276385b5-61ba-4c52-a013-ba32fdccafdc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.864570 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.864516    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/276385b5-61ba-4c52-a013-ba32fdccafdc-tmp\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.864570 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.864549    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/276385b5-61ba-4c52-a013-ba32fdccafdc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.950447 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.950413    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 18:43:16.950696 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.950675    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 18:43:16.950765 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.950682    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 18:43:16.950765 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.950712    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 18:43:16.950765 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.950737    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wzclj\""
Oct 30 18:43:16.950862 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.950801    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 18:43:16.965153 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.965126    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddkf8\" (UniqueName: \"kubernetes.io/projected/276385b5-61ba-4c52-a013-ba32fdccafdc-kube-api-access-ddkf8\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.965257 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.965156    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/276385b5-61ba-4c52-a013-ba32fdccafdc-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.965257 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.965174    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/276385b5-61ba-4c52-a013-ba32fdccafdc-manifests\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.965318 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.965286    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/276385b5-61ba-4c52-a013-ba32fdccafdc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.965356 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.965319    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/276385b5-61ba-4c52-a013-ba32fdccafdc-tmp\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.965410 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.965358    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/276385b5-61ba-4c52-a013-ba32fdccafdc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.965777 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.965752    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/276385b5-61ba-4c52-a013-ba32fdccafdc-manifests\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.967791 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.967770    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/276385b5-61ba-4c52-a013-ba32fdccafdc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.967791 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.967775    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/276385b5-61ba-4c52-a013-ba32fdccafdc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.967935 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.967909    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/276385b5-61ba-4c52-a013-ba32fdccafdc-tmp\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.968095 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.968074    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/276385b5-61ba-4c52-a013-ba32fdccafdc-metrics-cert\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:16.997261 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:16.997228    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"]
Oct 30 18:43:17.023748 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:17.023715    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddkf8\" (UniqueName: \"kubernetes.io/projected/276385b5-61ba-4c52-a013-ba32fdccafdc-kube-api-access-ddkf8\") pod \"opendatahub-operator-controller-manager-7cb69c88fb-dqpcm\" (UID: \"276385b5-61ba-4c52-a013-ba32fdccafdc\") " pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:17.100697 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:17.100610    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:17.595848 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:17.595817    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"]
Oct 30 18:43:17.601224 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:43:17.601196    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276385b5_61ba_4c52_a013_ba32fdccafdc.slice/crio-17c347f83e28c7e31bbcfe64a231bef4993bb147b9721f39e3a94b1047af35b9 WatchSource:0}: Error finding container 17c347f83e28c7e31bbcfe64a231bef4993bb147b9721f39e3a94b1047af35b9: Status 404 returned error can't find the container with id 17c347f83e28c7e31bbcfe64a231bef4993bb147b9721f39e3a94b1047af35b9
Oct 30 18:43:18.298382 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:18.298342    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm" event={"ID":"276385b5-61ba-4c52-a013-ba32fdccafdc","Type":"ContainerStarted","Data":"17c347f83e28c7e31bbcfe64a231bef4993bb147b9721f39e3a94b1047af35b9"}
Oct 30 18:43:20.160614 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:43:20.160570    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:43:22.316362 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:22.316328    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm" event={"ID":"276385b5-61ba-4c52-a013-ba32fdccafdc","Type":"ContainerStarted","Data":"a90223a8ceba5bdba30f0b0b76414649b4ae6bb02202acd567ded2ccc028b2f2"}
Oct 30 18:43:23.323119 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:23.323084    4267 generic.go:356] "Generic (PLEG): container finished" podID="276385b5-61ba-4c52-a013-ba32fdccafdc" containerID="a90223a8ceba5bdba30f0b0b76414649b4ae6bb02202acd567ded2ccc028b2f2" exitCode=0
Oct 30 18:43:23.323567 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:23.323123    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm" event={"ID":"276385b5-61ba-4c52-a013-ba32fdccafdc","Type":"ContainerDied","Data":"a90223a8ceba5bdba30f0b0b76414649b4ae6bb02202acd567ded2ccc028b2f2"}
Oct 30 18:43:24.329249 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:24.329219    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm" event={"ID":"276385b5-61ba-4c52-a013-ba32fdccafdc","Type":"ContainerStarted","Data":"e2892a55c1fc77deaee4587c8cd7ecce1d847e45c8954827f6d54e856774db51"}
Oct 30 18:43:24.329648 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:24.329335    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:24.554930 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:24.554823    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm" podStartSLOduration=3.04453627 podStartE2EDuration="8.554808229s" totalImagesPullingTime="5.027128858s" totalInitContainerRuntime="483.143101ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:24.504989807 +0000 UTC m=+295.392906907" watchObservedRunningTime="2026-10-30 18:43:24.554808229 +0000 UTC m=+295.442725317"
Oct 30 18:43:35.335857 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:35.335819    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7cb69c88fb-dqpcm"
Oct 30 18:43:40.183331 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.183295    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"]
Oct 30 18:43:40.187270 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.187249    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.237874 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.237832    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5599feb7-82b3-4fd6-a592-f7b671791e6d-tls-certs\") pod \"kube-auth-proxy-5fb96896c4-ph4kk\" (UID: \"5599feb7-82b3-4fd6-a592-f7b671791e6d\") " pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.237874 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.237871    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5599feb7-82b3-4fd6-a592-f7b671791e6d-tmp\") pod \"kube-auth-proxy-5fb96896c4-ph4kk\" (UID: \"5599feb7-82b3-4fd6-a592-f7b671791e6d\") " pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.238119 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.237965    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v7qq\" (UniqueName: \"kubernetes.io/projected/5599feb7-82b3-4fd6-a592-f7b671791e6d-kube-api-access-9v7qq\") pod \"kube-auth-proxy-5fb96896c4-ph4kk\" (UID: \"5599feb7-82b3-4fd6-a592-f7b671791e6d\") " pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.292518 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.292466    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 18:43:40.292756 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.292466    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 18:43:40.292756 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.292510    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4w8lc\""
Oct 30 18:43:40.292899 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.292510    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:43:40.292899 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.292578    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:43:40.294482 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.294459    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"]
Oct 30 18:43:40.338765 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.338730    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v7qq\" (UniqueName: \"kubernetes.io/projected/5599feb7-82b3-4fd6-a592-f7b671791e6d-kube-api-access-9v7qq\") pod \"kube-auth-proxy-5fb96896c4-ph4kk\" (UID: \"5599feb7-82b3-4fd6-a592-f7b671791e6d\") " pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.338949 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.338790    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5599feb7-82b3-4fd6-a592-f7b671791e6d-tls-certs\") pod \"kube-auth-proxy-5fb96896c4-ph4kk\" (UID: \"5599feb7-82b3-4fd6-a592-f7b671791e6d\") " pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.338949 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.338807    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5599feb7-82b3-4fd6-a592-f7b671791e6d-tmp\") pod \"kube-auth-proxy-5fb96896c4-ph4kk\" (UID: \"5599feb7-82b3-4fd6-a592-f7b671791e6d\") " pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.341069 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.341041    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5599feb7-82b3-4fd6-a592-f7b671791e6d-tmp\") pod \"kube-auth-proxy-5fb96896c4-ph4kk\" (UID: \"5599feb7-82b3-4fd6-a592-f7b671791e6d\") " pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.341309 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.341289    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5599feb7-82b3-4fd6-a592-f7b671791e6d-tls-certs\") pod \"kube-auth-proxy-5fb96896c4-ph4kk\" (UID: \"5599feb7-82b3-4fd6-a592-f7b671791e6d\") " pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.395069 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.395042    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v7qq\" (UniqueName: \"kubernetes.io/projected/5599feb7-82b3-4fd6-a592-f7b671791e6d-kube-api-access-9v7qq\") pod \"kube-auth-proxy-5fb96896c4-ph4kk\" (UID: \"5599feb7-82b3-4fd6-a592-f7b671791e6d\") " pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.496922 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.496834    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"
Oct 30 18:43:40.498196 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.498165    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"]
Oct 30 18:43:40.500049 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.500034    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.540052 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.540020    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93bb8f71-d869-4b4a-97c7-9502a2a0a292-metrics-cert\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.540202 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.540059    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8hp\" (UniqueName: \"kubernetes.io/projected/93bb8f71-d869-4b4a-97c7-9502a2a0a292-kube-api-access-7j8hp\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.540202 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.540138    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93bb8f71-d869-4b4a-97c7-9502a2a0a292-cert\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.540202 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.540173    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/93bb8f71-d869-4b4a-97c7-9502a2a0a292-manager-config\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.551985 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.551963    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 18:43:40.551985 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.551963    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 18:43:40.552198 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.551963    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 18:43:40.552198 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.552109    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 18:43:40.552198 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.552147    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:43:40.552314 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.552221    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tvjsk\""
Oct 30 18:43:40.559634 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.559564    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"]
Oct 30 18:43:40.641192 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.641162    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93bb8f71-d869-4b4a-97c7-9502a2a0a292-cert\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.641390 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.641201    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/93bb8f71-d869-4b4a-97c7-9502a2a0a292-manager-config\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.641390 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.641240    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93bb8f71-d869-4b4a-97c7-9502a2a0a292-metrics-cert\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.641390 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.641259    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8hp\" (UniqueName: \"kubernetes.io/projected/93bb8f71-d869-4b4a-97c7-9502a2a0a292-kube-api-access-7j8hp\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.641950 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.641925    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/93bb8f71-d869-4b4a-97c7-9502a2a0a292-manager-config\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.643661 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.643639    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/93bb8f71-d869-4b4a-97c7-9502a2a0a292-metrics-cert\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.643743 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.643687    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93bb8f71-d869-4b4a-97c7-9502a2a0a292-cert\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.698267 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.698233    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8hp\" (UniqueName: \"kubernetes.io/projected/93bb8f71-d869-4b4a-97c7-9502a2a0a292-kube-api-access-7j8hp\") pod \"lws-controller-manager-cf57f8b59-htgp7\" (UID: \"93bb8f71-d869-4b4a-97c7-9502a2a0a292\") " pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.822081 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.821995    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:40.866328 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.863765    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk"]
Oct 30 18:43:40.870181 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:40.870156    4267 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:43:41.206738 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:41.206714    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"]
Oct 30 18:43:41.209081 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:43:41.209051    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bb8f71_d869_4b4a_97c7_9502a2a0a292.slice/crio-75f0e6af44e221fb4a9c500a444060aa941b14cdebc8d71fb7c8bdbcb7d9cc77 WatchSource:0}: Error finding container 75f0e6af44e221fb4a9c500a444060aa941b14cdebc8d71fb7c8bdbcb7d9cc77: Status 404 returned error can't find the container with id 75f0e6af44e221fb4a9c500a444060aa941b14cdebc8d71fb7c8bdbcb7d9cc77
Oct 30 18:43:41.395285 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:41.395236    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7" event={"ID":"93bb8f71-d869-4b4a-97c7-9502a2a0a292","Type":"ContainerStarted","Data":"75f0e6af44e221fb4a9c500a444060aa941b14cdebc8d71fb7c8bdbcb7d9cc77"}
Oct 30 18:43:41.396396 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:41.396370    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk" event={"ID":"5599feb7-82b3-4fd6-a592-f7b671791e6d","Type":"ContainerStarted","Data":"50846854c659ad172609fc13360aea8601c275989fa3925d8a33df6ed18c106b"}
Oct 30 18:43:44.410916 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:44.410882    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk" event={"ID":"5599feb7-82b3-4fd6-a592-f7b671791e6d","Type":"ContainerStarted","Data":"e21a3cd9beeb2d56f94ea2d8beb57e44aaa5eb836f1957f855cacb5955e0ec60"}
Oct 30 18:43:44.525896 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:44.525835    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fb96896c4-ph4kk" podStartSLOduration=1.825596067 podStartE2EDuration="4.525815745s" totalImagesPullingTime="2.700219678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:44.524805332 +0000 UTC m=+315.412722421" watchObservedRunningTime="2026-10-30 18:43:44.525815745 +0000 UTC m=+315.413732835"
Oct 30 18:43:45.416246 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:45.416204    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7" event={"ID":"93bb8f71-d869-4b4a-97c7-9502a2a0a292","Type":"ContainerStarted","Data":"be2d03cb7349142834a1706b0856d9e4ec33492ba466318af4dd62a037f6055b"}
Oct 30 18:43:45.531588 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:45.531530    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7" podStartSLOduration=2.331021964 podStartE2EDuration="5.531512112s" totalImagesPullingTime="3.200490148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:45.52998731 +0000 UTC m=+316.417904399" watchObservedRunningTime="2026-10-30 18:43:45.531512112 +0000 UTC m=+316.419429204"
Oct 30 18:43:46.424177 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:46.424146    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:43:50.156319 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:43:50.156283    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:43:57.429959 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:43:57.429928    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-cf57f8b59-htgp7"
Oct 30 18:44:20.156298 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:44:20.156253    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:44:50.157319 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:44:50.157282    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:45:20.158639 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:45:20.158560    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:45:50.158591 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:45:50.158545    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:46:15.986100 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:15.986069    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"]
Oct 30 18:46:15.989387 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:15.989368    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.033535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.033481    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.033714 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.033616    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pjl\" (UniqueName: \"kubernetes.io/projected/7ab14388-0088-4c88-ab39-5938619b0cd1-kube-api-access-p4pjl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.033714 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.033658    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.091271 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.091227    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:46:16.091271 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.091227    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:46:16.091271 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.091238    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-z4hkm\""
Oct 30 18:46:16.092788 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.092770    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"]
Oct 30 18:46:16.093953 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.093934    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"]
Oct 30 18:46:16.097283 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.097262    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.134474 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.134436    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.134662 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.134486    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nz2\" (UniqueName: \"kubernetes.io/projected/f546ff17-366d-4524-9ae3-493d18e52199-kube-api-access-q5nz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.134662 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.134535    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.134662 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.134555    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pjl\" (UniqueName: \"kubernetes.io/projected/7ab14388-0088-4c88-ab39-5938619b0cd1-kube-api-access-p4pjl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.134662 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.134579    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.134662 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.134602    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.134893 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.134857    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.134932 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.134906    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.149468 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.149443    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-klj48\""
Oct 30 18:46:16.159922 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.159895    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"]
Oct 30 18:46:16.179470 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.179439    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"]
Oct 30 18:46:16.182785 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.182769    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.191589 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.191560    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pjl\" (UniqueName: \"kubernetes.io/projected/7ab14388-0088-4c88-ab39-5938619b0cd1-kube-api-access-p4pjl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.236252 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.236159    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nz2\" (UniqueName: \"kubernetes.io/projected/f546ff17-366d-4524-9ae3-493d18e52199-kube-api-access-q5nz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.236252 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.236209    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.236457 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.236373    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.236457 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.236416    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzs7b\" (UniqueName: \"kubernetes.io/projected/c749477c-5e62-4254-9d5d-95bbd0d9c325-kube-api-access-rzs7b\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.236582 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.236563    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.236623 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.236604    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.236666 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.236651    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.236874 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.236855    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.241121 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.241094    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"]
Oct 30 18:46:16.249377 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.249355    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-wjxrg\""
Oct 30 18:46:16.282437 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.282407    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"]
Oct 30 18:46:16.287382 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.287348    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.293286 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.293262    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nz2\" (UniqueName: \"kubernetes.io/projected/f546ff17-366d-4524-9ae3-493d18e52199-kube-api-access-q5nz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.304672 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.304650    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:16.337384 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.337352    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.337546 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.337400    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.337546 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.337428    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2d8x\" (UniqueName: \"kubernetes.io/projected/2af5687b-3bb3-4003-82af-4af18ae45f9f-kube-api-access-w2d8x\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.337546 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.337529    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.337667 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.337636    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.337701 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.337678    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzs7b\" (UniqueName: \"kubernetes.io/projected/c749477c-5e62-4254-9d5d-95bbd0d9c325-kube-api-access-rzs7b\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.337796 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.337775    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.337924 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.337910    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.344313 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.344247    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"]
Oct 30 18:46:16.349689 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.349663    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-jtptz\""
Oct 30 18:46:16.394967 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.394930    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzs7b\" (UniqueName: \"kubernetes.io/projected/c749477c-5e62-4254-9d5d-95bbd0d9c325-kube-api-access-rzs7b\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.412875 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.412841    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:16.438598 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.438566    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.438718 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.438623    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2d8x\" (UniqueName: \"kubernetes.io/projected/2af5687b-3bb3-4003-82af-4af18ae45f9f-kube-api-access-w2d8x\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.438718 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.438649    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.438921 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.438902    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.439008 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.438988    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.495809 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.495741    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2d8x\" (UniqueName: \"kubernetes.io/projected/2af5687b-3bb3-4003-82af-4af18ae45f9f-kube-api-access-w2d8x\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.499563 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.499540    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:16.610903 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.610867    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:16.676149 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.676123    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"]
Oct 30 18:46:16.677746 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:46:16.677721    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab14388_0088_4c88_ab39_5938619b0cd1.slice/crio-1ba7b9ec4af781035a74159e6611acf040f7701cea295b79710362edff2756de WatchSource:0}: Error finding container 1ba7b9ec4af781035a74159e6611acf040f7701cea295b79710362edff2756de: Status 404 returned error can't find the container with id 1ba7b9ec4af781035a74159e6611acf040f7701cea295b79710362edff2756de
Oct 30 18:46:16.783102 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.783076    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"]
Oct 30 18:46:16.827413 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:46:16.827375    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf546ff17_366d_4524_9ae3_493d18e52199.slice/crio-f4980cd774f207a726f80da34037b12e02985aa5a0edad6cdd65cd893ec63232 WatchSource:0}: Error finding container f4980cd774f207a726f80da34037b12e02985aa5a0edad6cdd65cd893ec63232: Status 404 returned error can't find the container with id f4980cd774f207a726f80da34037b12e02985aa5a0edad6cdd65cd893ec63232
Oct 30 18:46:16.867300 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.867194    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"]
Oct 30 18:46:16.869674 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:46:16.869644    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc749477c_5e62_4254_9d5d_95bbd0d9c325.slice/crio-2e77e069065b35c0826c7aefcc7d7a431b580d7a092bce859354829bd0217c1d WatchSource:0}: Error finding container 2e77e069065b35c0826c7aefcc7d7a431b580d7a092bce859354829bd0217c1d: Status 404 returned error can't find the container with id 2e77e069065b35c0826c7aefcc7d7a431b580d7a092bce859354829bd0217c1d
Oct 30 18:46:16.979044 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:16.978927    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"]
Oct 30 18:46:17.011726 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:17.011690    4267 generic.go:356] "Generic (PLEG): container finished" podID="c749477c-5e62-4254-9d5d-95bbd0d9c325" containerID="947565fe899998fff36e2e4ef23c2c700318f463874b01b65ee52b4fc2efc9a0" exitCode=0
Oct 30 18:46:17.012135 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:17.011768    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd" event={"ID":"c749477c-5e62-4254-9d5d-95bbd0d9c325","Type":"ContainerDied","Data":"947565fe899998fff36e2e4ef23c2c700318f463874b01b65ee52b4fc2efc9a0"}
Oct 30 18:46:17.012135 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:17.011826    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd" event={"ID":"c749477c-5e62-4254-9d5d-95bbd0d9c325","Type":"ContainerStarted","Data":"2e77e069065b35c0826c7aefcc7d7a431b580d7a092bce859354829bd0217c1d"}
Oct 30 18:46:17.013308 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:17.013287    4267 generic.go:356] "Generic (PLEG): container finished" podID="f546ff17-366d-4524-9ae3-493d18e52199" containerID="9a14e1869eea96eef2cc0aa64dc2ec1c09cf6c5a03f820768e2a0303213367ea" exitCode=0
Oct 30 18:46:17.013378 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:17.013315    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh" event={"ID":"f546ff17-366d-4524-9ae3-493d18e52199","Type":"ContainerDied","Data":"9a14e1869eea96eef2cc0aa64dc2ec1c09cf6c5a03f820768e2a0303213367ea"}
Oct 30 18:46:17.013378 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:17.013361    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh" event={"ID":"f546ff17-366d-4524-9ae3-493d18e52199","Type":"ContainerStarted","Data":"f4980cd774f207a726f80da34037b12e02985aa5a0edad6cdd65cd893ec63232"}
Oct 30 18:46:17.014983 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:17.014929    4267 generic.go:356] "Generic (PLEG): container finished" podID="7ab14388-0088-4c88-ab39-5938619b0cd1" containerID="b0587f27e73cc74b32defc00f2e2812325f76fb8f33b87d78d3a76650233fc63" exitCode=0
Oct 30 18:46:17.015077 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:17.015007    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk" event={"ID":"7ab14388-0088-4c88-ab39-5938619b0cd1","Type":"ContainerDied","Data":"b0587f27e73cc74b32defc00f2e2812325f76fb8f33b87d78d3a76650233fc63"}
Oct 30 18:46:17.015077 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:17.015041    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk" event={"ID":"7ab14388-0088-4c88-ab39-5938619b0cd1","Type":"ContainerStarted","Data":"1ba7b9ec4af781035a74159e6611acf040f7701cea295b79710362edff2756de"}
Oct 30 18:46:17.028331 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:46:17.028306    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af5687b_3bb3_4003_82af_4af18ae45f9f.slice/crio-24c036e945df67e65055a6595ad0624281ef2b83a14896f4e91987f0077e8a16 WatchSource:0}: Error finding container 24c036e945df67e65055a6595ad0624281ef2b83a14896f4e91987f0077e8a16: Status 404 returned error can't find the container with id 24c036e945df67e65055a6595ad0624281ef2b83a14896f4e91987f0077e8a16
Oct 30 18:46:18.021115 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:18.021082    4267 generic.go:356] "Generic (PLEG): container finished" podID="2af5687b-3bb3-4003-82af-4af18ae45f9f" containerID="c1095146fef6b6bed37e3a3ef1cd7da193cb4b2c9cd78a25093fddec4980a725" exitCode=0
Oct 30 18:46:18.021514 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:18.021131    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp" event={"ID":"2af5687b-3bb3-4003-82af-4af18ae45f9f","Type":"ContainerDied","Data":"c1095146fef6b6bed37e3a3ef1cd7da193cb4b2c9cd78a25093fddec4980a725"}
Oct 30 18:46:18.021514 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:18.021166    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp" event={"ID":"2af5687b-3bb3-4003-82af-4af18ae45f9f","Type":"ContainerStarted","Data":"24c036e945df67e65055a6595ad0624281ef2b83a14896f4e91987f0077e8a16"}
Oct 30 18:46:19.027672 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:19.027643    4267 generic.go:356] "Generic (PLEG): container finished" podID="7ab14388-0088-4c88-ab39-5938619b0cd1" containerID="5f6d063988da3c70cb1b87e3c95b1034355577684646f0455c4a9e13ad40ab1f" exitCode=0
Oct 30 18:46:19.028087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:19.027729    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk" event={"ID":"7ab14388-0088-4c88-ab39-5938619b0cd1","Type":"ContainerDied","Data":"5f6d063988da3c70cb1b87e3c95b1034355577684646f0455c4a9e13ad40ab1f"}
Oct 30 18:46:19.030299 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:19.030267    4267 generic.go:356] "Generic (PLEG): container finished" podID="c749477c-5e62-4254-9d5d-95bbd0d9c325" containerID="863c1107c0bcd38659e2b219f61b19f3d4fd49792a0ef957686a5126b48a1dd5" exitCode=0
Oct 30 18:46:19.030391 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:19.030360    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd" event={"ID":"c749477c-5e62-4254-9d5d-95bbd0d9c325","Type":"ContainerDied","Data":"863c1107c0bcd38659e2b219f61b19f3d4fd49792a0ef957686a5126b48a1dd5"}
Oct 30 18:46:19.032925 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:19.032906    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh" event={"ID":"f546ff17-366d-4524-9ae3-493d18e52199","Type":"ContainerStarted","Data":"e79d02a773a5d0488cf2ca2bbfa29c2d51dca9e599660070ec67320ed15c78fa"}
Oct 30 18:46:20.040587 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:20.040557    4267 generic.go:356] "Generic (PLEG): container finished" podID="c749477c-5e62-4254-9d5d-95bbd0d9c325" containerID="e68865bcea060993192debe8f05b00227dfb3a0edd1e7036b58a5476f8ac58ec" exitCode=0
Oct 30 18:46:20.040898 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:20.040639    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd" event={"ID":"c749477c-5e62-4254-9d5d-95bbd0d9c325","Type":"ContainerDied","Data":"e68865bcea060993192debe8f05b00227dfb3a0edd1e7036b58a5476f8ac58ec"}
Oct 30 18:46:20.042241 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:20.042215    4267 generic.go:356] "Generic (PLEG): container finished" podID="f546ff17-366d-4524-9ae3-493d18e52199" containerID="e79d02a773a5d0488cf2ca2bbfa29c2d51dca9e599660070ec67320ed15c78fa" exitCode=0
Oct 30 18:46:20.042342 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:20.042293    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh" event={"ID":"f546ff17-366d-4524-9ae3-493d18e52199","Type":"ContainerDied","Data":"e79d02a773a5d0488cf2ca2bbfa29c2d51dca9e599660070ec67320ed15c78fa"}
Oct 30 18:46:20.044667 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:20.044646    4267 generic.go:356] "Generic (PLEG): container finished" podID="7ab14388-0088-4c88-ab39-5938619b0cd1" containerID="51b2c745e2b8aa7419e55e491cfa4d6f1f91f8a3695b28a3960cb3d63d44eb42" exitCode=0
Oct 30 18:46:20.044750 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:20.044705    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk" event={"ID":"7ab14388-0088-4c88-ab39-5938619b0cd1","Type":"ContainerDied","Data":"51b2c745e2b8aa7419e55e491cfa4d6f1f91f8a3695b28a3960cb3d63d44eb42"}
Oct 30 18:46:20.157229 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:46:20.157185    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:46:21.050732 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.050696    4267 generic.go:356] "Generic (PLEG): container finished" podID="f546ff17-366d-4524-9ae3-493d18e52199" containerID="c01c753bd18fb3f9c0e15d004e495aa16a61c808127a3d833bcbf1769b0e23cd" exitCode=0
Oct 30 18:46:21.051179 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.050777    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh" event={"ID":"f546ff17-366d-4524-9ae3-493d18e52199","Type":"ContainerDied","Data":"c01c753bd18fb3f9c0e15d004e495aa16a61c808127a3d833bcbf1769b0e23cd"}
Oct 30 18:46:21.052555 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.052534    4267 generic.go:356] "Generic (PLEG): container finished" podID="2af5687b-3bb3-4003-82af-4af18ae45f9f" containerID="19f9fb57c032f71e56fdbb5eff90533763ba8cf7bcd4287e31f8d5e7e6b0978a" exitCode=0
Oct 30 18:46:21.052681 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.052611    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp" event={"ID":"2af5687b-3bb3-4003-82af-4af18ae45f9f","Type":"ContainerDied","Data":"19f9fb57c032f71e56fdbb5eff90533763ba8cf7bcd4287e31f8d5e7e6b0978a"}
Oct 30 18:46:21.208155 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.208127    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:21.211258 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.211237    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:21.277530 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.277469    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-util\" (UniqueName: \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-util\") pod \"7ab14388-0088-4c88-ab39-5938619b0cd1\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") "
Oct 30 18:46:21.277707 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.277543    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c749477c-5e62-4254-9d5d-95bbd0d9c325-kube-api-access-rzs7b\" (UniqueName: \"kubernetes.io/projected/c749477c-5e62-4254-9d5d-95bbd0d9c325-kube-api-access-rzs7b\") pod \"c749477c-5e62-4254-9d5d-95bbd0d9c325\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") "
Oct 30 18:46:21.277707 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.277622    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-bundle\") pod \"7ab14388-0088-4c88-ab39-5938619b0cd1\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") "
Oct 30 18:46:21.277707 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.277690    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-util\" (UniqueName: \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-util\") pod \"c749477c-5e62-4254-9d5d-95bbd0d9c325\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") "
Oct 30 18:46:21.277939 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.277720    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab14388-0088-4c88-ab39-5938619b0cd1-kube-api-access-p4pjl\" (UniqueName: \"kubernetes.io/projected/7ab14388-0088-4c88-ab39-5938619b0cd1-kube-api-access-p4pjl\") pod \"7ab14388-0088-4c88-ab39-5938619b0cd1\" (UID: \"7ab14388-0088-4c88-ab39-5938619b0cd1\") "
Oct 30 18:46:21.277939 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.277750    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-bundle\") pod \"c749477c-5e62-4254-9d5d-95bbd0d9c325\" (UID: \"c749477c-5e62-4254-9d5d-95bbd0d9c325\") "
Oct 30 18:46:21.278552 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.278523    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-bundle" pod "c749477c-5e62-4254-9d5d-95bbd0d9c325" (UID: "c749477c-5e62-4254-9d5d-95bbd0d9c325"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:46:21.279060 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.279023    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-bundle" pod "7ab14388-0088-4c88-ab39-5938619b0cd1" (UID: "7ab14388-0088-4c88-ab39-5938619b0cd1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:46:21.280150 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.280122    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c749477c-5e62-4254-9d5d-95bbd0d9c325-kube-api-access-rzs7b" pod "c749477c-5e62-4254-9d5d-95bbd0d9c325" (UID: "c749477c-5e62-4254-9d5d-95bbd0d9c325"). InnerVolumeSpecName "kube-api-access-rzs7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:21.281291 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.281259    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab14388-0088-4c88-ab39-5938619b0cd1-kube-api-access-p4pjl" pod "7ab14388-0088-4c88-ab39-5938619b0cd1" (UID: "7ab14388-0088-4c88-ab39-5938619b0cd1"). InnerVolumeSpecName "kube-api-access-p4pjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:21.283823 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.283799    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-util" pod "c749477c-5e62-4254-9d5d-95bbd0d9c325" (UID: "c749477c-5e62-4254-9d5d-95bbd0d9c325"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:46:21.285486 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.285461    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-util" pod "7ab14388-0088-4c88-ab39-5938619b0cd1" (UID: "7ab14388-0088-4c88-ab39-5938619b0cd1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:46:21.379285 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.379189    4267 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-bundle\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:21.379285 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.379221    4267 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-util\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:21.379285 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.379233    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4pjl\" (UniqueName: \"kubernetes.io/projected/7ab14388-0088-4c88-ab39-5938619b0cd1-kube-api-access-p4pjl\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:21.379285 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.379245    4267 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c749477c-5e62-4254-9d5d-95bbd0d9c325-bundle\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:21.379285 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.379256    4267 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ab14388-0088-4c88-ab39-5938619b0cd1-util\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:21.379285 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:21.379267    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzs7b\" (UniqueName: \"kubernetes.io/projected/c749477c-5e62-4254-9d5d-95bbd0d9c325-kube-api-access-rzs7b\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:22.059609 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.059577    4267 generic.go:356] "Generic (PLEG): container finished" podID="2af5687b-3bb3-4003-82af-4af18ae45f9f" containerID="e2fce6199eaf3820d9adbe919bb4826665b7b1736979d864cac64f80e47c81d1" exitCode=0
Oct 30 18:46:22.060081 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.059662    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp" event={"ID":"2af5687b-3bb3-4003-82af-4af18ae45f9f","Type":"ContainerDied","Data":"e2fce6199eaf3820d9adbe919bb4826665b7b1736979d864cac64f80e47c81d1"}
Oct 30 18:46:22.061628 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.061603    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd"
Oct 30 18:46:22.061628 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.061618    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmjzwd" event={"ID":"c749477c-5e62-4254-9d5d-95bbd0d9c325","Type":"ContainerDied","Data":"2e77e069065b35c0826c7aefcc7d7a431b580d7a092bce859354829bd0217c1d"}
Oct 30 18:46:22.061790 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.061657    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e77e069065b35c0826c7aefcc7d7a431b580d7a092bce859354829bd0217c1d"
Oct 30 18:46:22.063639 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.063613    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk" event={"ID":"7ab14388-0088-4c88-ab39-5938619b0cd1","Type":"ContainerDied","Data":"1ba7b9ec4af781035a74159e6611acf040f7701cea295b79710362edff2756de"}
Oct 30 18:46:22.063639 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.063637    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a57bpk"
Oct 30 18:46:22.063791 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.063657    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ba7b9ec4af781035a74159e6611acf040f7701cea295b79710362edff2756de"
Oct 30 18:46:22.191779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.191752    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:22.286951 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.286920    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f546ff17-366d-4524-9ae3-493d18e52199-kube-api-access-q5nz2\" (UniqueName: \"kubernetes.io/projected/f546ff17-366d-4524-9ae3-493d18e52199-kube-api-access-q5nz2\") pod \"f546ff17-366d-4524-9ae3-493d18e52199\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") "
Oct 30 18:46:22.287097 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.286958    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-bundle\") pod \"f546ff17-366d-4524-9ae3-493d18e52199\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") "
Oct 30 18:46:22.287097 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.287008    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-util\" (UniqueName: \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-util\") pod \"f546ff17-366d-4524-9ae3-493d18e52199\" (UID: \"f546ff17-366d-4524-9ae3-493d18e52199\") "
Oct 30 18:46:22.287552 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.287519    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-bundle" pod "f546ff17-366d-4524-9ae3-493d18e52199" (UID: "f546ff17-366d-4524-9ae3-493d18e52199"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:46:22.289105 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.289086    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f546ff17-366d-4524-9ae3-493d18e52199-kube-api-access-q5nz2" pod "f546ff17-366d-4524-9ae3-493d18e52199" (UID: "f546ff17-366d-4524-9ae3-493d18e52199"). InnerVolumeSpecName "kube-api-access-q5nz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:22.292853 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.292822    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-util" pod "f546ff17-366d-4524-9ae3-493d18e52199" (UID: "f546ff17-366d-4524-9ae3-493d18e52199"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:46:22.388441 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.388354    4267 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-util\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:22.388441 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.388395    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5nz2\" (UniqueName: \"kubernetes.io/projected/f546ff17-366d-4524-9ae3-493d18e52199-kube-api-access-q5nz2\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:22.388441 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:22.388406    4267 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f546ff17-366d-4524-9ae3-493d18e52199-bundle\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:23.069806 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.069768    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh" event={"ID":"f546ff17-366d-4524-9ae3-493d18e52199","Type":"ContainerDied","Data":"f4980cd774f207a726f80da34037b12e02985aa5a0edad6cdd65cd893ec63232"}
Oct 30 18:46:23.069806 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.069793    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799mhh"
Oct 30 18:46:23.070358 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.069821    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4980cd774f207a726f80da34037b12e02985aa5a0edad6cdd65cd893ec63232"
Oct 30 18:46:23.196158 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.196130    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:23.294560 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.294526    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-util\" (UniqueName: \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-util\") pod \"2af5687b-3bb3-4003-82af-4af18ae45f9f\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") "
Oct 30 18:46:23.294736 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.294582    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af5687b-3bb3-4003-82af-4af18ae45f9f-kube-api-access-w2d8x\" (UniqueName: \"kubernetes.io/projected/2af5687b-3bb3-4003-82af-4af18ae45f9f-kube-api-access-w2d8x\") pod \"2af5687b-3bb3-4003-82af-4af18ae45f9f\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") "
Oct 30 18:46:23.294736 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.294632    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-bundle\") pod \"2af5687b-3bb3-4003-82af-4af18ae45f9f\" (UID: \"2af5687b-3bb3-4003-82af-4af18ae45f9f\") "
Oct 30 18:46:23.295081 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.295051    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-bundle" pod "2af5687b-3bb3-4003-82af-4af18ae45f9f" (UID: "2af5687b-3bb3-4003-82af-4af18ae45f9f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:46:23.296817 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.296795    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af5687b-3bb3-4003-82af-4af18ae45f9f-kube-api-access-w2d8x" pod "2af5687b-3bb3-4003-82af-4af18ae45f9f" (UID: "2af5687b-3bb3-4003-82af-4af18ae45f9f"). InnerVolumeSpecName "kube-api-access-w2d8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:23.299151 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.299127    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-util" pod "2af5687b-3bb3-4003-82af-4af18ae45f9f" (UID: "2af5687b-3bb3-4003-82af-4af18ae45f9f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:46:23.395918 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.395879    4267 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-util\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:23.395918 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.395912    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2d8x\" (UniqueName: \"kubernetes.io/projected/2af5687b-3bb3-4003-82af-4af18ae45f9f-kube-api-access-w2d8x\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:23.395918 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:23.395923    4267 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2af5687b-3bb3-4003-82af-4af18ae45f9f-bundle\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:24.075983 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:24.075948    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp" event={"ID":"2af5687b-3bb3-4003-82af-4af18ae45f9f","Type":"ContainerDied","Data":"24c036e945df67e65055a6595ad0624281ef2b83a14896f4e91987f0077e8a16"}
Oct 30 18:46:24.076385 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:24.075993    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gn6zp"
Oct 30 18:46:24.076385 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:24.075995    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24c036e945df67e65055a6595ad0624281ef2b83a14896f4e91987f0077e8a16"
Oct 30 18:46:45.669513 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669406    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"]
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669811    4267 cpu_manager.go:414] "containerMap: removing container" podUID="f546ff17-366d-4524-9ae3-493d18e52199" containerName="extract"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669829    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f546ff17-366d-4524-9ae3-493d18e52199" containerName="extract"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669853    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c749477c-5e62-4254-9d5d-95bbd0d9c325" containerName="pull"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669862    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c749477c-5e62-4254-9d5d-95bbd0d9c325" containerName="pull"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669870    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c749477c-5e62-4254-9d5d-95bbd0d9c325" containerName="extract"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669878    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c749477c-5e62-4254-9d5d-95bbd0d9c325" containerName="extract"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669890    4267 cpu_manager.go:414] "containerMap: removing container" podUID="7ab14388-0088-4c88-ab39-5938619b0cd1" containerName="extract"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669897    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab14388-0088-4c88-ab39-5938619b0cd1" containerName="extract"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669914    4267 cpu_manager.go:414] "containerMap: removing container" podUID="2af5687b-3bb3-4003-82af-4af18ae45f9f" containerName="util"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669920    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af5687b-3bb3-4003-82af-4af18ae45f9f" containerName="util"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669933    4267 cpu_manager.go:414] "containerMap: removing container" podUID="7ab14388-0088-4c88-ab39-5938619b0cd1" containerName="util"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669939    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab14388-0088-4c88-ab39-5938619b0cd1" containerName="util"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669949    4267 cpu_manager.go:414] "containerMap: removing container" podUID="7ab14388-0088-4c88-ab39-5938619b0cd1" containerName="pull"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669956    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab14388-0088-4c88-ab39-5938619b0cd1" containerName="pull"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669970    4267 cpu_manager.go:414] "containerMap: removing container" podUID="f546ff17-366d-4524-9ae3-493d18e52199" containerName="pull"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669977    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f546ff17-366d-4524-9ae3-493d18e52199" containerName="pull"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669987    4267 cpu_manager.go:414] "containerMap: removing container" podUID="2af5687b-3bb3-4003-82af-4af18ae45f9f" containerName="extract"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.669994    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af5687b-3bb3-4003-82af-4af18ae45f9f" containerName="extract"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.670006    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c749477c-5e62-4254-9d5d-95bbd0d9c325" containerName="util"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.670012    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c749477c-5e62-4254-9d5d-95bbd0d9c325" containerName="util"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.670023    4267 cpu_manager.go:414] "containerMap: removing container" podUID="f546ff17-366d-4524-9ae3-493d18e52199" containerName="util"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.670030    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f546ff17-366d-4524-9ae3-493d18e52199" containerName="util"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.670041    4267 cpu_manager.go:414] "containerMap: removing container" podUID="2af5687b-3bb3-4003-82af-4af18ae45f9f" containerName="pull"
Oct 30 18:46:45.670087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.670048    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af5687b-3bb3-4003-82af-4af18ae45f9f" containerName="pull"
Oct 30 18:46:45.671166 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.670126    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c749477c-5e62-4254-9d5d-95bbd0d9c325" containerName="extract"
Oct 30 18:46:45.671166 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.670139    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab14388-0088-4c88-ab39-5938619b0cd1" containerName="extract"
Oct 30 18:46:45.671166 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.670150    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af5687b-3bb3-4003-82af-4af18ae45f9f" containerName="extract"
Oct 30 18:46:45.671166 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.670159    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="f546ff17-366d-4524-9ae3-493d18e52199" containerName="extract"
Oct 30 18:46:45.672984 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.672964    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:45.730055 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.730016    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 18:46:45.731683 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.731659    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"]
Oct 30 18:46:45.779624 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.779588    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 18:46:45.779624 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.779600    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 18:46:45.779624 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.779619    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 18:46:45.779853 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.779648    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hdxfm\""
Oct 30 18:46:45.786143 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.786118    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9n45v\" (UID: \"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:45.786233 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.786180    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9n45v\" (UID: \"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:45.786268 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.786251    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxp7j\" (UniqueName: \"kubernetes.io/projected/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-kube-api-access-gxp7j\") pod \"kuadrant-console-plugin-66c4fb575d-9n45v\" (UID: \"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:45.887312 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.887273    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9n45v\" (UID: \"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:45.887549 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.887348    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9n45v\" (UID: \"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:45.887549 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.887371    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxp7j\" (UniqueName: \"kubernetes.io/projected/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-kube-api-access-gxp7j\") pod \"kuadrant-console-plugin-66c4fb575d-9n45v\" (UID: \"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:45.887549 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:46:45.887445    4267 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Oct 30 18:46:45.887727 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:46:45.887560    4267 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-plugin-serving-cert podName:22c77e4c-cf9d-470b-8ab2-c1cad5221b8c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:46.387539307 +0000 UTC m=+497.275456391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-9n45v" (UID: "22c77e4c-cf9d-470b-8ab2-c1cad5221b8c") : secret "plugin-serving-cert" not found
Oct 30 18:46:45.888014 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.887997    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9n45v\" (UID: \"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:45.946336 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:45.946306    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxp7j\" (UniqueName: \"kubernetes.io/projected/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-kube-api-access-gxp7j\") pod \"kuadrant-console-plugin-66c4fb575d-9n45v\" (UID: \"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:46.390630 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:46.390542    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9n45v\" (UID: \"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:46.392957 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:46.392932    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/22c77e4c-cf9d-470b-8ab2-c1cad5221b8c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9n45v\" (UID: \"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:46.583468 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:46.583431    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"
Oct 30 18:46:46.950933 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:46.950910    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v"]
Oct 30 18:46:46.952782 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:46:46.952752    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22c77e4c_cf9d_470b_8ab2_c1cad5221b8c.slice/crio-e794cb8004e052c0fd745536a172165b5d13e86c38a24faa6551e0668412a94c WatchSource:0}: Error finding container e794cb8004e052c0fd745536a172165b5d13e86c38a24faa6551e0668412a94c: Status 404 returned error can't find the container with id e794cb8004e052c0fd745536a172165b5d13e86c38a24faa6551e0668412a94c
Oct 30 18:46:47.173774 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:47.173732    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v" event={"ID":"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c","Type":"ContainerStarted","Data":"e794cb8004e052c0fd745536a172165b5d13e86c38a24faa6551e0668412a94c"}
Oct 30 18:46:50.161608 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:46:50.161561    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:46:53.203670 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.203633    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v" event={"ID":"22c77e4c-cf9d-470b-8ab2-c1cad5221b8c","Type":"ContainerStarted","Data":"4edcbd17b20fa35ed2dbeab90489eaa9aedfc5058c95b7335d4e0b928c0521ac"}
Oct 30 18:46:53.317850 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.317793    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9n45v" podStartSLOduration=2.543362051 podStartE2EDuration="8.317778196s" totalImagesPullingTime="5.774416145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:53.316770164 +0000 UTC m=+504.204687249" watchObservedRunningTime="2026-10-30 18:46:53.317778196 +0000 UTC m=+504.205695285"
Oct 30 18:46:53.368468 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.368438    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"]
Oct 30 18:46:53.371894 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.371872    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:53.422078 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.422039    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"]
Oct 30 18:46:53.423805 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.423781    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l677f\""
Oct 30 18:46:53.447040 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.447010    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20cf0eeb-ce61-4007-9ccd-d87d1653749e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-qknhz\" (UID: \"20cf0eeb-ce61-4007-9ccd-d87d1653749e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:53.447202 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.447064    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vj5\" (UniqueName: \"kubernetes.io/projected/20cf0eeb-ce61-4007-9ccd-d87d1653749e-kube-api-access-44vj5\") pod \"kuadrant-operator-controller-manager-956956bc4-qknhz\" (UID: \"20cf0eeb-ce61-4007-9ccd-d87d1653749e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:53.548339 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.548243    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vj5\" (UniqueName: \"kubernetes.io/projected/20cf0eeb-ce61-4007-9ccd-d87d1653749e-kube-api-access-44vj5\") pod \"kuadrant-operator-controller-manager-956956bc4-qknhz\" (UID: \"20cf0eeb-ce61-4007-9ccd-d87d1653749e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:53.548339 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.548299    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20cf0eeb-ce61-4007-9ccd-d87d1653749e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-qknhz\" (UID: \"20cf0eeb-ce61-4007-9ccd-d87d1653749e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:53.548762 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.548739    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20cf0eeb-ce61-4007-9ccd-d87d1653749e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-qknhz\" (UID: \"20cf0eeb-ce61-4007-9ccd-d87d1653749e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:53.606555 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.606521    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vj5\" (UniqueName: \"kubernetes.io/projected/20cf0eeb-ce61-4007-9ccd-d87d1653749e-kube-api-access-44vj5\") pod \"kuadrant-operator-controller-manager-956956bc4-qknhz\" (UID: \"20cf0eeb-ce61-4007-9ccd-d87d1653749e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:53.684732 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:53.684691    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:54.175574 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:54.175546    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"]
Oct 30 18:46:54.177387 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:46:54.177359    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cf0eeb_ce61_4007_9ccd_d87d1653749e.slice/crio-9b1cba8053a4e7ca6b13b03f8a164a7df7887c23460210a1ac75d5a3d6293ba2 WatchSource:0}: Error finding container 9b1cba8053a4e7ca6b13b03f8a164a7df7887c23460210a1ac75d5a3d6293ba2: Status 404 returned error can't find the container with id 9b1cba8053a4e7ca6b13b03f8a164a7df7887c23460210a1ac75d5a3d6293ba2
Oct 30 18:46:54.213535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:54.213484    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz" event={"ID":"20cf0eeb-ce61-4007-9ccd-d87d1653749e","Type":"ContainerStarted","Data":"9b1cba8053a4e7ca6b13b03f8a164a7df7887c23460210a1ac75d5a3d6293ba2"}
Oct 30 18:46:55.163252 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:55.163218    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"]
Oct 30 18:46:55.164707 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:55.164686    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"]
Oct 30 18:46:59.239454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:59.239400    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz" podUID="20cf0eeb-ce61-4007-9ccd-d87d1653749e" containerName="manager" containerID="cri-o://7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d" gracePeriod=2
Oct 30 18:46:59.291742 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:46:59.291705    4267 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-qknhz\" is forbidden: User \"system:node:ip-10-0-131-68.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-68.us-west-1.compute.internal' and this object" podUID="20cf0eeb-ce61-4007-9ccd-d87d1653749e" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:59.527844 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:59.527817    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:59.578610 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:46:59.578568    4267 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-qknhz\" is forbidden: User \"system:node:ip-10-0-131-68.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-68.us-west-1.compute.internal' and this object" podUID="20cf0eeb-ce61-4007-9ccd-d87d1653749e" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:46:59.599945 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:59.599919    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20cf0eeb-ce61-4007-9ccd-d87d1653749e-kube-api-access-44vj5\" (UniqueName: \"kubernetes.io/projected/20cf0eeb-ce61-4007-9ccd-d87d1653749e-kube-api-access-44vj5\") pod \"20cf0eeb-ce61-4007-9ccd-d87d1653749e\" (UID: \"20cf0eeb-ce61-4007-9ccd-d87d1653749e\") "
Oct 30 18:46:59.600063 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:59.599968    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20cf0eeb-ce61-4007-9ccd-d87d1653749e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20cf0eeb-ce61-4007-9ccd-d87d1653749e-extensions-socket-volume\") pod \"20cf0eeb-ce61-4007-9ccd-d87d1653749e\" (UID: \"20cf0eeb-ce61-4007-9ccd-d87d1653749e\") "
Oct 30 18:46:59.600270 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:59.600239    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20cf0eeb-ce61-4007-9ccd-d87d1653749e-extensions-socket-volume" pod "20cf0eeb-ce61-4007-9ccd-d87d1653749e" (UID: "20cf0eeb-ce61-4007-9ccd-d87d1653749e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:46:59.602215 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:59.602193    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cf0eeb-ce61-4007-9ccd-d87d1653749e-kube-api-access-44vj5" pod "20cf0eeb-ce61-4007-9ccd-d87d1653749e" (UID: "20cf0eeb-ce61-4007-9ccd-d87d1653749e"). InnerVolumeSpecName "kube-api-access-44vj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:59.701437 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:59.701392    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44vj5\" (UniqueName: \"kubernetes.io/projected/20cf0eeb-ce61-4007-9ccd-d87d1653749e-kube-api-access-44vj5\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:46:59.701437 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:46:59.701426    4267 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20cf0eeb-ce61-4007-9ccd-d87d1653749e-extensions-socket-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:47:00.245510 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:00.245462    4267 generic.go:356] "Generic (PLEG): container finished" podID="20cf0eeb-ce61-4007-9ccd-d87d1653749e" containerID="7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d" exitCode=2
Oct 30 18:47:00.246010 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:00.245527    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:47:00.246010 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:00.245557    4267 scope.go:122] "RemoveContainer" containerID="7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d"
Oct 30 18:47:00.253983 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:00.253966    4267 scope.go:122] "RemoveContainer" containerID="7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d"
Oct 30 18:47:00.254279 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:47:00.254262    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d\": container with ID starting with 7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d not found: ID does not exist" containerID="7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d"
Oct 30 18:47:00.254315 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:00.254290    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d"} err="failed to get container status \"7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d\": rpc error: code = NotFound desc = could not find container \"7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d\": container with ID starting with 7e3c96aca4a0642aaf022cef378dff7975a4fa5664f9288270a71ace8ea2fe9d not found: ID does not exist"
Oct 30 18:47:00.296756 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:47:00.296715    4267 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-qknhz\" is forbidden: User \"system:node:ip-10-0-131-68.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-68.us-west-1.compute.internal' and this object" podUID="20cf0eeb-ce61-4007-9ccd-d87d1653749e" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:47:00.347472 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:47:00.347429    4267 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-qknhz\" is forbidden: User \"system:node:ip-10-0-131-68.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-68.us-west-1.compute.internal' and this object" podUID="20cf0eeb-ce61-4007-9ccd-d87d1653749e" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-qknhz"
Oct 30 18:47:01.219318 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:01.219282    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20cf0eeb-ce61-4007-9ccd-d87d1653749e" path="/var/lib/kubelet/pods/20cf0eeb-ce61-4007-9ccd-d87d1653749e/volumes"
Oct 30 18:47:08.149859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.149825    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"]
Oct 30 18:47:08.150346 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.150248    4267 cpu_manager.go:414] "containerMap: removing container" podUID="20cf0eeb-ce61-4007-9ccd-d87d1653749e" containerName="manager"
Oct 30 18:47:08.150346 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.150267    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20cf0eeb-ce61-4007-9ccd-d87d1653749e" containerName="manager"
Oct 30 18:47:08.150346 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.150345    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="20cf0eeb-ce61-4007-9ccd-d87d1653749e" containerName="manager"
Oct 30 18:47:08.154832 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.154809    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 18:47:08.168441 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.168414    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"]
Oct 30 18:47:08.215851 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.215824    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l677f\""
Oct 30 18:47:08.266858 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.266821    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-b4gdh\" (UID: \"3c757ca6-2bb6-4274-8ee3-40a16a01ca29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 18:47:08.267042 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.266942    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpffl\" (UniqueName: \"kubernetes.io/projected/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-kube-api-access-dpffl\") pod \"kuadrant-operator-controller-manager-57bb9bfff-b4gdh\" (UID: \"3c757ca6-2bb6-4274-8ee3-40a16a01ca29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 18:47:08.367876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.367832    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpffl\" (UniqueName: \"kubernetes.io/projected/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-kube-api-access-dpffl\") pod \"kuadrant-operator-controller-manager-57bb9bfff-b4gdh\" (UID: \"3c757ca6-2bb6-4274-8ee3-40a16a01ca29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 18:47:08.367876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.367896    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-b4gdh\" (UID: \"3c757ca6-2bb6-4274-8ee3-40a16a01ca29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 18:47:08.368379 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.368356    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-b4gdh\" (UID: \"3c757ca6-2bb6-4274-8ee3-40a16a01ca29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 18:47:08.425102 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.425036    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpffl\" (UniqueName: \"kubernetes.io/projected/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-kube-api-access-dpffl\") pod \"kuadrant-operator-controller-manager-57bb9bfff-b4gdh\" (UID: \"3c757ca6-2bb6-4274-8ee3-40a16a01ca29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 18:47:08.467687 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.467654    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 18:47:08.891473 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:08.891450    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"]
Oct 30 18:47:08.893680 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:47:08.893648    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c757ca6_2bb6_4274_8ee3_40a16a01ca29.slice/crio-68343a645a690d88810d3f00acce33830d0769a31e365e95e27b6ec9dc767bb1 WatchSource:0}: Error finding container 68343a645a690d88810d3f00acce33830d0769a31e365e95e27b6ec9dc767bb1: Status 404 returned error can't find the container with id 68343a645a690d88810d3f00acce33830d0769a31e365e95e27b6ec9dc767bb1
Oct 30 18:47:09.285204 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:09.285168    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh" event={"ID":"3c757ca6-2bb6-4274-8ee3-40a16a01ca29","Type":"ContainerStarted","Data":"142bdc818342cab7170b0bf6829a6c8ff61aa0c4a1658b26e1b7bcf6dad5ec79"}
Oct 30 18:47:09.285606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:09.285220    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh" event={"ID":"3c757ca6-2bb6-4274-8ee3-40a16a01ca29","Type":"ContainerStarted","Data":"68343a645a690d88810d3f00acce33830d0769a31e365e95e27b6ec9dc767bb1"}
Oct 30 18:47:09.285606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:09.285272    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 18:47:09.455106 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:09.455051    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh" podStartSLOduration=1.455034621 podStartE2EDuration="1.455034621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:09.4548534 +0000 UTC m=+520.342770491" watchObservedRunningTime="2026-10-30 18:47:09.455034621 +0000 UTC m=+520.342951710"
Oct 30 18:47:20.157800 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:47:20.157762    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:47:20.291629 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:20.291601    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 18:47:45.342017 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:45.341980    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snxvt"]
Oct 30 18:47:45.344105 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:45.344085    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-snxvt"
Oct 30 18:47:45.398883 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:45.398850    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snxvt"]
Oct 30 18:47:45.445670 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:45.445644    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nv74j\""
Oct 30 18:47:45.470806 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:45.470771    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47slh\" (UniqueName: \"kubernetes.io/projected/a4f747ba-2f72-4fb1-bee3-03b3779a1605-kube-api-access-47slh\") pod \"authorino-56c6c9ddd5-snxvt\" (UID: \"a4f747ba-2f72-4fb1-bee3-03b3779a1605\") " pod="kuadrant-system/authorino-56c6c9ddd5-snxvt"
Oct 30 18:47:45.571540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:45.571474    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47slh\" (UniqueName: \"kubernetes.io/projected/a4f747ba-2f72-4fb1-bee3-03b3779a1605-kube-api-access-47slh\") pod \"authorino-56c6c9ddd5-snxvt\" (UID: \"a4f747ba-2f72-4fb1-bee3-03b3779a1605\") " pod="kuadrant-system/authorino-56c6c9ddd5-snxvt"
Oct 30 18:47:45.633063 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:45.632992    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47slh\" (UniqueName: \"kubernetes.io/projected/a4f747ba-2f72-4fb1-bee3-03b3779a1605-kube-api-access-47slh\") pod \"authorino-56c6c9ddd5-snxvt\" (UID: \"a4f747ba-2f72-4fb1-bee3-03b3779a1605\") " pod="kuadrant-system/authorino-56c6c9ddd5-snxvt"
Oct 30 18:47:45.655272 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:45.655246    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-snxvt"
Oct 30 18:47:46.020082 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:46.020057    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snxvt"]
Oct 30 18:47:46.021799 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:47:46.021764    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4f747ba_2f72_4fb1_bee3_03b3779a1605.slice/crio-cdf10aa8eb1aca33c39835cf2acf96528c69905440901dac8f932057123d9069 WatchSource:0}: Error finding container cdf10aa8eb1aca33c39835cf2acf96528c69905440901dac8f932057123d9069: Status 404 returned error can't find the container with id cdf10aa8eb1aca33c39835cf2acf96528c69905440901dac8f932057123d9069
Oct 30 18:47:46.450161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:46.450130    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-snxvt" event={"ID":"a4f747ba-2f72-4fb1-bee3-03b3779a1605","Type":"ContainerStarted","Data":"cdf10aa8eb1aca33c39835cf2acf96528c69905440901dac8f932057123d9069"}
Oct 30 18:47:49.465445 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:49.465399    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-snxvt" event={"ID":"a4f747ba-2f72-4fb1-bee3-03b3779a1605","Type":"ContainerStarted","Data":"2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2"}
Oct 30 18:47:49.578735 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:47:49.578680    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-snxvt" podStartSLOduration=1.872651066 podStartE2EDuration="4.578665222s" totalImagesPullingTime="2.706014156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:49.576970801 +0000 UTC m=+560.464887891" watchObservedRunningTime="2026-10-30 18:47:49.578665222 +0000 UTC m=+560.466582311"
Oct 30 18:47:50.158118 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:47:50.158079    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:48:20.157255 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:48:20.157164    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:48:20.850166 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:20.850129    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7lhph"]
Oct 30 18:48:20.853825 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:20.853807    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7lhph"
Oct 30 18:48:20.910168 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:20.910131    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7lhph"]
Oct 30 18:48:20.954790 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:20.954754    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c752b\" (UniqueName: \"kubernetes.io/projected/75f63b08-d36a-425b-957c-77a9cb6dbae6-kube-api-access-c752b\") pod \"authorino-6f6944bcdd-7lhph\" (UID: \"75f63b08-d36a-425b-957c-77a9cb6dbae6\") " pod="kuadrant-system/authorino-6f6944bcdd-7lhph"
Oct 30 18:48:21.056697 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:21.056641    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c752b\" (UniqueName: \"kubernetes.io/projected/75f63b08-d36a-425b-957c-77a9cb6dbae6-kube-api-access-c752b\") pod \"authorino-6f6944bcdd-7lhph\" (UID: \"75f63b08-d36a-425b-957c-77a9cb6dbae6\") " pod="kuadrant-system/authorino-6f6944bcdd-7lhph"
Oct 30 18:48:21.114279 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:21.114202    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c752b\" (UniqueName: \"kubernetes.io/projected/75f63b08-d36a-425b-957c-77a9cb6dbae6-kube-api-access-c752b\") pod \"authorino-6f6944bcdd-7lhph\" (UID: \"75f63b08-d36a-425b-957c-77a9cb6dbae6\") " pod="kuadrant-system/authorino-6f6944bcdd-7lhph"
Oct 30 18:48:21.165657 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:21.165623    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7lhph"
Oct 30 18:48:21.196056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:21.196023    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7lhph"]
Oct 30 18:48:21.534957 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:21.534915    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7lhph"]
Oct 30 18:48:21.537942 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:48:21.537917    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75f63b08_d36a_425b_957c_77a9cb6dbae6.slice/crio-8705fcf91f080bf24640243097c17a61d55ed0f5c4920e47574c00e0dfa14772 WatchSource:0}: Error finding container 8705fcf91f080bf24640243097c17a61d55ed0f5c4920e47574c00e0dfa14772: Status 404 returned error can't find the container with id 8705fcf91f080bf24640243097c17a61d55ed0f5c4920e47574c00e0dfa14772
Oct 30 18:48:21.606447 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:21.606398    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7lhph" event={"ID":"75f63b08-d36a-425b-957c-77a9cb6dbae6","Type":"ContainerStarted","Data":"8705fcf91f080bf24640243097c17a61d55ed0f5c4920e47574c00e0dfa14772"}
Oct 30 18:48:23.615701 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:23.615609    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7lhph" event={"ID":"75f63b08-d36a-425b-957c-77a9cb6dbae6","Type":"ContainerStarted","Data":"cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b"}
Oct 30 18:48:23.616141 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:23.615760    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-7lhph" podUID="75f63b08-d36a-425b-957c-77a9cb6dbae6" containerName="authorino" containerID="cri-o://cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b" gracePeriod=30
Oct 30 18:48:23.728203 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:23.728145    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-7lhph" podStartSLOduration=2.232028905 podStartE2EDuration="3.728127977s" totalImagesPullingTime="1.496099072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:23.726306884 +0000 UTC m=+594.614223973" watchObservedRunningTime="2026-10-30 18:48:23.728127977 +0000 UTC m=+594.616045066"
Oct 30 18:48:23.860850 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:23.860825    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7lhph"
Oct 30 18:48:23.984223 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:23.984183    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f63b08-d36a-425b-957c-77a9cb6dbae6-kube-api-access-c752b\" (UniqueName: \"kubernetes.io/projected/75f63b08-d36a-425b-957c-77a9cb6dbae6-kube-api-access-c752b\") pod \"75f63b08-d36a-425b-957c-77a9cb6dbae6\" (UID: \"75f63b08-d36a-425b-957c-77a9cb6dbae6\") "
Oct 30 18:48:23.986547 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:23.986511    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f63b08-d36a-425b-957c-77a9cb6dbae6-kube-api-access-c752b" pod "75f63b08-d36a-425b-957c-77a9cb6dbae6" (UID: "75f63b08-d36a-425b-957c-77a9cb6dbae6"). InnerVolumeSpecName "kube-api-access-c752b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:24.085334 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.085288    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c752b\" (UniqueName: \"kubernetes.io/projected/75f63b08-d36a-425b-957c-77a9cb6dbae6-kube-api-access-c752b\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:48:24.126859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.126818    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snxvt"]
Oct 30 18:48:24.127032 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.127010    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-snxvt" podUID="a4f747ba-2f72-4fb1-bee3-03b3779a1605" containerName="authorino" containerID="cri-o://2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2" gracePeriod=30
Oct 30 18:48:24.379642 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.379616    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-snxvt"
Oct 30 18:48:24.488479 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.488424    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f747ba-2f72-4fb1-bee3-03b3779a1605-kube-api-access-47slh\" (UniqueName: \"kubernetes.io/projected/a4f747ba-2f72-4fb1-bee3-03b3779a1605-kube-api-access-47slh\") pod \"a4f747ba-2f72-4fb1-bee3-03b3779a1605\" (UID: \"a4f747ba-2f72-4fb1-bee3-03b3779a1605\") "
Oct 30 18:48:24.490536 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.490486    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f747ba-2f72-4fb1-bee3-03b3779a1605-kube-api-access-47slh" pod "a4f747ba-2f72-4fb1-bee3-03b3779a1605" (UID: "a4f747ba-2f72-4fb1-bee3-03b3779a1605"). InnerVolumeSpecName "kube-api-access-47slh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:24.589732 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.589625    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47slh\" (UniqueName: \"kubernetes.io/projected/a4f747ba-2f72-4fb1-bee3-03b3779a1605-kube-api-access-47slh\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:48:24.621584 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.621543    4267 generic.go:356] "Generic (PLEG): container finished" podID="75f63b08-d36a-425b-957c-77a9cb6dbae6" containerID="cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b" exitCode=0
Oct 30 18:48:24.622069 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.621589    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7lhph"
Oct 30 18:48:24.622069 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.621632    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7lhph" event={"ID":"75f63b08-d36a-425b-957c-77a9cb6dbae6","Type":"ContainerDied","Data":"cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b"}
Oct 30 18:48:24.622069 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.621688    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7lhph" event={"ID":"75f63b08-d36a-425b-957c-77a9cb6dbae6","Type":"ContainerDied","Data":"8705fcf91f080bf24640243097c17a61d55ed0f5c4920e47574c00e0dfa14772"}
Oct 30 18:48:24.622069 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.621706    4267 scope.go:122] "RemoveContainer" containerID="cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b"
Oct 30 18:48:24.623013 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.622986    4267 generic.go:356] "Generic (PLEG): container finished" podID="a4f747ba-2f72-4fb1-bee3-03b3779a1605" containerID="2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2" exitCode=0
Oct 30 18:48:24.623135 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.623019    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-snxvt"
Oct 30 18:48:24.623135 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.623121    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-snxvt" event={"ID":"a4f747ba-2f72-4fb1-bee3-03b3779a1605","Type":"ContainerDied","Data":"2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2"}
Oct 30 18:48:24.623228 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.623153    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-snxvt" event={"ID":"a4f747ba-2f72-4fb1-bee3-03b3779a1605","Type":"ContainerDied","Data":"cdf10aa8eb1aca33c39835cf2acf96528c69905440901dac8f932057123d9069"}
Oct 30 18:48:24.631669 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.631645    4267 scope.go:122] "RemoveContainer" containerID="cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b"
Oct 30 18:48:24.631964 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:48:24.631945    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b\": container with ID starting with cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b not found: ID does not exist" containerID="cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b"
Oct 30 18:48:24.632018 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.631975    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b"} err="failed to get container status \"cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b\": rpc error: code = NotFound desc = could not find container \"cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b\": container with ID starting with cb71e17c04cd3eef338d7277c7e3484f3fe1f69f6b3987883a662e912138177b not found: ID does not exist"
Oct 30 18:48:24.632018 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.631993    4267 scope.go:122] "RemoveContainer" containerID="2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2"
Oct 30 18:48:24.639661 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.639642    4267 scope.go:122] "RemoveContainer" containerID="2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2"
Oct 30 18:48:24.639934 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:48:24.639914    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2\": container with ID starting with 2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2 not found: ID does not exist" containerID="2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2"
Oct 30 18:48:24.639988 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.639945    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2"} err="failed to get container status \"2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2\": rpc error: code = NotFound desc = could not find container \"2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2\": container with ID starting with 2ea3e4dc2b8faf1bdc662b49899b3be8d9814cd7ce44a98eb898a55c5a7cfad2 not found: ID does not exist"
Oct 30 18:48:24.785852 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.785811    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7lhph"]
Oct 30 18:48:24.789762 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.789725    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7lhph"]
Oct 30 18:48:24.902147 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.902057    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snxvt"]
Oct 30 18:48:24.908336 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:24.908309    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-snxvt"]
Oct 30 18:48:25.218786 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:25.218755    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f63b08-d36a-425b-957c-77a9cb6dbae6" path="/var/lib/kubelet/pods/75f63b08-d36a-425b-957c-77a9cb6dbae6/volumes"
Oct 30 18:48:25.219084 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:48:25.219071    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f747ba-2f72-4fb1-bee3-03b3779a1605" path="/var/lib/kubelet/pods/a4f747ba-2f72-4fb1-bee3-03b3779a1605/volumes"
Oct 30 18:48:50.157823 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:48:50.157786    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:49:20.158894 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:49:20.158853    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:49:50.156515 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:49:50.156420    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:50:15.197962 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.197907    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"]
Oct 30 18:50:15.198583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.198234    4267 cpu_manager.go:414] "containerMap: removing container" podUID="a4f747ba-2f72-4fb1-bee3-03b3779a1605" containerName="authorino"
Oct 30 18:50:15.198583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.198245    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f747ba-2f72-4fb1-bee3-03b3779a1605" containerName="authorino"
Oct 30 18:50:15.198583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.198256    4267 cpu_manager.go:414] "containerMap: removing container" podUID="75f63b08-d36a-425b-957c-77a9cb6dbae6" containerName="authorino"
Oct 30 18:50:15.198583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.198262    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f63b08-d36a-425b-957c-77a9cb6dbae6" containerName="authorino"
Oct 30 18:50:15.198583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.198311    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f747ba-2f72-4fb1-bee3-03b3779a1605" containerName="authorino"
Oct 30 18:50:15.198583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.198319    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f63b08-d36a-425b-957c-77a9cb6dbae6" containerName="authorino"
Oct 30 18:50:15.201364 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.201348    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.253446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.253407    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 18:50:15.264766 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.264735    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.264941 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.264776    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.264941 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.264855    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8bz\" (UniqueName: \"kubernetes.io/projected/46d1bbb4-c192-4a1e-a791-cd06f0976714-kube-api-access-zm8bz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.265061 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.264966    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.265061 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.265018    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.265150 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.265060    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d1bbb4-c192-4a1e-a791-cd06f0976714-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.302841 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.302807    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 18:50:15.303006 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.302859    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ntlts\""
Oct 30 18:50:15.303006 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.302879    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 18:50:15.304821 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.304798    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"]
Oct 30 18:50:15.366054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.366017    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.366054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.366059    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.366313 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.366080    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d1bbb4-c192-4a1e-a791-cd06f0976714-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.366313 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.366103    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.366313 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.366137    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.366313 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.366181    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8bz\" (UniqueName: \"kubernetes.io/projected/46d1bbb4-c192-4a1e-a791-cd06f0976714-kube-api-access-zm8bz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.366565 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.366539    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.366619 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.366571    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.366735 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.366708    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.368486 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.368467    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46d1bbb4-c192-4a1e-a791-cd06f0976714-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.368679 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.368663    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46d1bbb4-c192-4a1e-a791-cd06f0976714-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.423896 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.423861    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8bz\" (UniqueName: \"kubernetes.io/projected/46d1bbb4-c192-4a1e-a791-cd06f0976714-kube-api-access-zm8bz\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg\" (UID: \"46d1bbb4-c192-4a1e-a791-cd06f0976714\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.513857 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.513770    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:15.882310 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.882286    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"]
Oct 30 18:50:15.883811 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:50:15.883783    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d1bbb4_c192_4a1e_a791_cd06f0976714.slice/crio-09c73267dd95bf2d95ffffa11d77d74b7b0c537b0363f0c037aa0dbeae72f2bc WatchSource:0}: Error finding container 09c73267dd95bf2d95ffffa11d77d74b7b0c537b0363f0c037aa0dbeae72f2bc: Status 404 returned error can't find the container with id 09c73267dd95bf2d95ffffa11d77d74b7b0c537b0363f0c037aa0dbeae72f2bc
Oct 30 18:50:15.885105 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:15.885089    4267 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:50:16.106397 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:16.106361    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg" event={"ID":"46d1bbb4-c192-4a1e-a791-cd06f0976714","Type":"ContainerStarted","Data":"09c73267dd95bf2d95ffffa11d77d74b7b0c537b0363f0c037aa0dbeae72f2bc"}
Oct 30 18:50:19.087965 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.087928    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"]
Oct 30 18:50:19.091209 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.091189    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.123876 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.123843    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg" event={"ID":"46d1bbb4-c192-4a1e-a791-cd06f0976714","Type":"ContainerStarted","Data":"6df27fe7281695d1083e78aba81991c3a1cc55e6fb7566ab5cd4d0727afdd94c"}
Oct 30 18:50:19.161448 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.161425    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 18:50:19.194609 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.194584    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"]
Oct 30 18:50:19.198001 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.197968    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.198135 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.198027    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.198197 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.198164    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.198245 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.198209    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnqfp\" (UniqueName: \"kubernetes.io/projected/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-kube-api-access-wnqfp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.198332 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.198305    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.198368 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.198352    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.299350 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.299310    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.299538 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.299361    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.299538 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.299388    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.299538 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.299412    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnqfp\" (UniqueName: \"kubernetes.io/projected/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-kube-api-access-wnqfp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.299538 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.299446    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.299538 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.299461    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.299836 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.299819    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.299891 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.299849    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.299981 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.299907    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.301780 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.301758    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.302056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.302039    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.321035 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.320991    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg" podStartSLOduration=2.143247541 podStartE2EDuration="4.320979033s" totalImagesPullingTime="2.177731492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:19.319907953 +0000 UTC m=+710.207825039" watchObservedRunningTime="2026-10-30 18:50:19.320979033 +0000 UTC m=+710.208896141"
Oct 30 18:50:19.355170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.355111    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnqfp\" (UniqueName: \"kubernetes.io/projected/26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553-kube-api-access-wnqfp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp\" (UID: \"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.403356 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.403320    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:19.514140 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.514102    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:19.785407 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:19.785368    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"]
Oct 30 18:50:19.788242 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:50:19.788218    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e4fdf8_054d_4f5f_bb3a_9bdb3cc3d553.slice/crio-3f11a88ada913c957334617b5ab0c09ad2026b041455d4d03ce7cee59eefaaa9 WatchSource:0}: Error finding container 3f11a88ada913c957334617b5ab0c09ad2026b041455d4d03ce7cee59eefaaa9: Status 404 returned error can't find the container with id 3f11a88ada913c957334617b5ab0c09ad2026b041455d4d03ce7cee59eefaaa9
Oct 30 18:50:20.130029 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:20.129942    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp" event={"ID":"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553","Type":"ContainerStarted","Data":"3f11a88ada913c957334617b5ab0c09ad2026b041455d4d03ce7cee59eefaaa9"}
Oct 30 18:50:20.156538 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:50:20.156479    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:50:21.136884 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:21.136849    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp" event={"ID":"26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553","Type":"ContainerStarted","Data":"58a4e47fa2b0b632f32e183ef704aa45f44135081dd414378c68a3433211f697"}
Oct 30 18:50:21.259308 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:21.259248    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp" podStartSLOduration=1.663187054 podStartE2EDuration="2.25923248s" totalImagesPullingTime="596.045426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:21.259191783 +0000 UTC m=+712.147108873" watchObservedRunningTime="2026-10-30 18:50:21.25923248 +0000 UTC m=+712.147149568"
Oct 30 18:50:21.404031 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:21.403937    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:25.514113 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:25.514073    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:25.526974 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:25.526948    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:26.171464 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.171437    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg"
Oct 30 18:50:26.371124 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.371089    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"]
Oct 30 18:50:26.374593 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.374574    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.427045 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.426957    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 18:50:26.435890 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.435859    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"]
Oct 30 18:50:26.459224 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.459193    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e108a863-9bb1-46e7-a1ca-93c23dc91d92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.459400 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.459232    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.459400 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.459255    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.459400 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.459391    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kxhf\" (UniqueName: \"kubernetes.io/projected/e108a863-9bb1-46e7-a1ca-93c23dc91d92-kube-api-access-6kxhf\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.459556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.459430    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.459556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.459550    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.560074 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.560042    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kxhf\" (UniqueName: \"kubernetes.io/projected/e108a863-9bb1-46e7-a1ca-93c23dc91d92-kube-api-access-6kxhf\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.560565 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.560082    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.560565 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.560144    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.560565 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.560183    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e108a863-9bb1-46e7-a1ca-93c23dc91d92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.560565 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.560215    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.560565 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.560242    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.560809 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.560582    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.560809 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.560661    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.560809 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.560677    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.562722 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.562697    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e108a863-9bb1-46e7-a1ca-93c23dc91d92-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.563015 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.562993    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e108a863-9bb1-46e7-a1ca-93c23dc91d92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.618289 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.618248    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kxhf\" (UniqueName: \"kubernetes.io/projected/e108a863-9bb1-46e7-a1ca-93c23dc91d92-kube-api-access-6kxhf\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb\" (UID: \"e108a863-9bb1-46e7-a1ca-93c23dc91d92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:26.685893 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:26.685845    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:27.061071 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:27.061044    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"]
Oct 30 18:50:27.063050 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:50:27.063013    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode108a863_9bb1_46e7_a1ca_93c23dc91d92.slice/crio-0486ab2682c361ed701700f41c53bc317c4437ba7d996e2330d6361e2de031bb WatchSource:0}: Error finding container 0486ab2682c361ed701700f41c53bc317c4437ba7d996e2330d6361e2de031bb: Status 404 returned error can't find the container with id 0486ab2682c361ed701700f41c53bc317c4437ba7d996e2330d6361e2de031bb
Oct 30 18:50:27.166919 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:27.166882    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" event={"ID":"e108a863-9bb1-46e7-a1ca-93c23dc91d92","Type":"ContainerStarted","Data":"0486ab2682c361ed701700f41c53bc317c4437ba7d996e2330d6361e2de031bb"}
Oct 30 18:50:28.174298 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:28.174206    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" event={"ID":"e108a863-9bb1-46e7-a1ca-93c23dc91d92","Type":"ContainerStarted","Data":"5ef65e29454b6d02c8896510f037a969bcc0184a2fde5194681b8a048071422b"}
Oct 30 18:50:28.287967 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:28.287917    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podStartSLOduration=1.66653551 podStartE2EDuration="2.287903792s" totalImagesPullingTime="621.368282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:28.287659803 +0000 UTC m=+719.175576891" watchObservedRunningTime="2026-10-30 18:50:28.287903792 +0000 UTC m=+719.175820881"
Oct 30 18:50:28.686150 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:28.686109    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:29.403922 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:29.403885    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:29.416456 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:29.416426    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:30.194291 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:30.194252    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp"
Oct 30 18:50:36.686961 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:36.686926    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:36.700508 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:36.700465    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:37.226181 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:37.226140    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 18:50:40.828802 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.828759    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"]
Oct 30 18:50:40.832104 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.832084    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:40.884725 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.884700    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 18:50:40.934010 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.933984    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qwc8z\""
Oct 30 18:50:40.937026 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.937003    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"]
Oct 30 18:50:40.980715 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.980682    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:40.980894 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.980745    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:40.980894 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.980830    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:40.980894 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.980884    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:40.981066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.980905    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:40.981066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.980927    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:40.981066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.981029    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63104ff-ed9d-4970-b1fe-06d50d28209f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:40.981066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.981049    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79tf5\" (UniqueName: \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-kube-api-access-79tf5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:40.981231 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:40.981109    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082019 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.081935    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082019 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.081974    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082019 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.081989    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082019 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082018    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082352 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082045    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082352 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082092    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63104ff-ed9d-4970-b1fe-06d50d28209f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082352 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082115    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79tf5\" (UniqueName: \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-kube-api-access-79tf5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082352 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082241    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082352 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082293    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082628 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082413    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082628 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082475    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082765 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082741    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082862 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082763    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.082922 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.082867    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63104ff-ed9d-4970-b1fe-06d50d28209f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.084397 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.084373    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.084620 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.084603    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.138809 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.138784    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.138997 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.138974    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79tf5\" (UniqueName: \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-kube-api-access-79tf5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-92mng\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.144648 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.144626    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:41.568007 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:41.567974    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"]
Oct 30 18:50:41.569190 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:50:41.569157    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63104ff_ed9d_4970_b1fe_06d50d28209f.slice/crio-174f8b8535fe6baff40aacbdeb9b32855d877ed6ee11d6d77bf687ad260b0f2a WatchSource:0}: Error finding container 174f8b8535fe6baff40aacbdeb9b32855d877ed6ee11d6d77bf687ad260b0f2a: Status 404 returned error can't find the container with id 174f8b8535fe6baff40aacbdeb9b32855d877ed6ee11d6d77bf687ad260b0f2a
Oct 30 18:50:42.237308 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:42.237274    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" event={"ID":"a63104ff-ed9d-4970-b1fe-06d50d28209f","Type":"ContainerStarted","Data":"174f8b8535fe6baff40aacbdeb9b32855d877ed6ee11d6d77bf687ad260b0f2a"}
Oct 30 18:50:46.234766 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:46.234729    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:50:46.235219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:46.234796    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:50:46.235219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:46.234825    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:50:47.263980 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:47.263942    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" event={"ID":"a63104ff-ed9d-4970-b1fe-06d50d28209f","Type":"ContainerStarted","Data":"94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb"}
Oct 30 18:50:47.483159 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:47.483099    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" podStartSLOduration=2.820906714 podStartE2EDuration="7.483086419s" totalImagesPullingTime="4.662179705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:47.43203253 +0000 UTC m=+738.319949643" watchObservedRunningTime="2026-10-30 18:50:47.483086419 +0000 UTC m=+738.371003507"
Oct 30 18:50:48.145351 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:48.145307    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:48.146943 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:48.146917    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-92mng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:50:48.147078 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:48.146979    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:50:49.145436 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:49.145396    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-92mng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:50:49.145864 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:49.145470    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:50:50.145057 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:50.145021    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-92mng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:50:50.145225 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:50.145104    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:50:50.163270 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:50:50.163230    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:50:51.145845 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:51.145811    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-92mng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:50:51.146030 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:51.145869    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:50:52.146070 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:52.146024    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-92mng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:50:52.146597 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:52.146088    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:50:53.145004 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:53.144968    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-92mng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 18:50:53.145191 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:53.145029    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 18:50:55.186620 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:55.186585    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-92mng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:50:55.187103 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:55.186644    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:50:55.832994 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:55.832950    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"]
Oct 30 18:50:55.838410 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:55.838382    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:55.939510 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:55.939456    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"]
Oct 30 18:50:56.014830 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.014791    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.015024 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.014854    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.015024 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.014968    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.015024 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.015013    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.015201 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.015096    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t997c\" (UniqueName: \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-kube-api-access-t997c\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.015201 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.015147    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.015291 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.015228    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a54a5660-e61e-48fd-9d91-33e37acff5b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.015341 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.015302    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.015430 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.015362    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.116931 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.116846    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.116931 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.116914    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.117147 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.116942    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.117147 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.116973    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.117147 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.117000    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t997c\" (UniqueName: \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-kube-api-access-t997c\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.117147 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.117037    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.117147 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.117087    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a54a5660-e61e-48fd-9d91-33e37acff5b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.117147 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.117130    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.117436 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.117178    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.117627 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.117579    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.118000 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.117972    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a54a5660-e61e-48fd-9d91-33e37acff5b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.118356 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.118337    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.118356 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.118341    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.118461 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.118447    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.119861 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.119827    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.120210 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.120186    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.145447 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.145408    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:56.168374 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.168337    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:56.173462 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.173434    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t997c\" (UniqueName: \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-kube-api-access-t997c\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.173674 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.173655    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-mctkp\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.305713 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.305688    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:50:56.455907 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.455873    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:56.884631 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.884604    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"]
Oct 30 18:50:56.887855 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:50:56.887825    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54a5660_e61e_48fd_9d91_33e37acff5b4.slice/crio-d1465965ad41afb04c846e049a60de2a21edc7d4548119c5184ea2dd32f3cd4c WatchSource:0}: Error finding container d1465965ad41afb04c846e049a60de2a21edc7d4548119c5184ea2dd32f3cd4c: Status 404 returned error can't find the container with id d1465965ad41afb04c846e049a60de2a21edc7d4548119c5184ea2dd32f3cd4c
Oct 30 18:50:56.890868 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.890835    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:50:56.890955 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.890906    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:50:56.890955 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:56.890935    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:50:57.182461 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.182383    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84f7b9c6bb-t78t4"]
Oct 30 18:50:57.186024 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.185999    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84f7b9c6bb-t78t4"
Oct 30 18:50:57.240274 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.240231    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nv74j\""
Oct 30 18:50:57.240473 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.240373    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 18:50:57.242642 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.242618    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84f7b9c6bb-t78t4"]
Oct 30 18:50:57.312106 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.312059    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" event={"ID":"a54a5660-e61e-48fd-9d91-33e37acff5b4","Type":"ContainerStarted","Data":"550aef35600f09143f4333aa4a7e77df3691abc58943c5fff28198426902e82d"}
Oct 30 18:50:57.312625 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.312133    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" event={"ID":"a54a5660-e61e-48fd-9d91-33e37acff5b4","Type":"ContainerStarted","Data":"d1465965ad41afb04c846e049a60de2a21edc7d4548119c5184ea2dd32f3cd4c"}
Oct 30 18:50:57.327899 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.327863    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e593c4e1-00ce-4f98-acbd-a1a0beb50b07-tls-cert\") pod \"authorino-84f7b9c6bb-t78t4\" (UID: \"e593c4e1-00ce-4f98-acbd-a1a0beb50b07\") " pod="kuadrant-system/authorino-84f7b9c6bb-t78t4"
Oct 30 18:50:57.327899 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.327902    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gksm2\" (UniqueName: \"kubernetes.io/projected/e593c4e1-00ce-4f98-acbd-a1a0beb50b07-kube-api-access-gksm2\") pod \"authorino-84f7b9c6bb-t78t4\" (UID: \"e593c4e1-00ce-4f98-acbd-a1a0beb50b07\") " pod="kuadrant-system/authorino-84f7b9c6bb-t78t4"
Oct 30 18:50:57.436688 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.436640    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e593c4e1-00ce-4f98-acbd-a1a0beb50b07-tls-cert\") pod \"authorino-84f7b9c6bb-t78t4\" (UID: \"e593c4e1-00ce-4f98-acbd-a1a0beb50b07\") " pod="kuadrant-system/authorino-84f7b9c6bb-t78t4"
Oct 30 18:50:57.436899 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.436705    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gksm2\" (UniqueName: \"kubernetes.io/projected/e593c4e1-00ce-4f98-acbd-a1a0beb50b07-kube-api-access-gksm2\") pod \"authorino-84f7b9c6bb-t78t4\" (UID: \"e593c4e1-00ce-4f98-acbd-a1a0beb50b07\") " pod="kuadrant-system/authorino-84f7b9c6bb-t78t4"
Oct 30 18:50:57.438508 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.438438    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" podStartSLOduration=2.43842486 podStartE2EDuration="2.43842486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:57.436685686 +0000 UTC m=+748.324602789" watchObservedRunningTime="2026-10-30 18:50:57.43842486 +0000 UTC m=+748.326341948"
Oct 30 18:50:57.442751 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.442724    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e593c4e1-00ce-4f98-acbd-a1a0beb50b07-tls-cert\") pod \"authorino-84f7b9c6bb-t78t4\" (UID: \"e593c4e1-00ce-4f98-acbd-a1a0beb50b07\") " pod="kuadrant-system/authorino-84f7b9c6bb-t78t4"
Oct 30 18:50:57.456406 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.456384    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:50:57.457684 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.457661    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mctkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:50:57.457801 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.457707    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:50:57.493431 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.493400    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gksm2\" (UniqueName: \"kubernetes.io/projected/e593c4e1-00ce-4f98-acbd-a1a0beb50b07-kube-api-access-gksm2\") pod \"authorino-84f7b9c6bb-t78t4\" (UID: \"e593c4e1-00ce-4f98-acbd-a1a0beb50b07\") " pod="kuadrant-system/authorino-84f7b9c6bb-t78t4"
Oct 30 18:50:57.501172 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.501144    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84f7b9c6bb-t78t4"
Oct 30 18:50:57.897393 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:57.897369    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84f7b9c6bb-t78t4"]
Oct 30 18:50:57.899067 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:50:57.899025    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode593c4e1_00ce_4f98_acbd_a1a0beb50b07.slice/crio-57ed6da57f2c6cc42f499d46e0fb36baaaf3c85f16039d348b601015a28751c7 WatchSource:0}: Error finding container 57ed6da57f2c6cc42f499d46e0fb36baaaf3c85f16039d348b601015a28751c7: Status 404 returned error can't find the container with id 57ed6da57f2c6cc42f499d46e0fb36baaaf3c85f16039d348b601015a28751c7
Oct 30 18:50:58.344523 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:58.344471    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84f7b9c6bb-t78t4" event={"ID":"e593c4e1-00ce-4f98-acbd-a1a0beb50b07","Type":"ContainerStarted","Data":"57ed6da57f2c6cc42f499d46e0fb36baaaf3c85f16039d348b601015a28751c7"}
Oct 30 18:50:58.456700 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:58.456668    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mctkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:50:58.456869 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:58.456721    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:50:59.457012 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:59.456980    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mctkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:50:59.457427 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:50:59.457033    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:51:00.355010 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:00.354977    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84f7b9c6bb-t78t4" event={"ID":"e593c4e1-00ce-4f98-acbd-a1a0beb50b07","Type":"ContainerStarted","Data":"26ba142cfcf7eaae6be9834f04834fdfa90c62c4a4c59f7639241e88fdcf5a03"}
Oct 30 18:51:00.456889 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:00.456838    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mctkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:51:00.457250 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:00.456896    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:51:00.468905 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:00.468856    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-84f7b9c6bb-t78t4" podStartSLOduration=1.787641357 podStartE2EDuration="3.468838324s" totalImagesPullingTime="1.681196967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:00.468046818 +0000 UTC m=+751.355963910" watchObservedRunningTime="2026-10-30 18:51:00.468838324 +0000 UTC m=+751.356755416"
Oct 30 18:51:01.457030 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:01.456995    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mctkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:51:01.457247 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:01.457226    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:51:02.456831 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:02.456791    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mctkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:51:02.457229 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:02.456852    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:51:03.457411 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:03.457352    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mctkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:51:03.457941 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:03.457424    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:51:04.456415 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:04.456379    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-mctkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 18:51:04.456671 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:04.456442    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 18:51:05.465234 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:05.465196    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:51:05.465868 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:05.465847    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:51:05.466602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:05.466580    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:51:10.848577 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.848532    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"]
Oct 30 18:51:10.852088 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.852062    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:10.956970 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.956940    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"]
Oct 30 18:51:10.962101 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.962067    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:10.962236 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.962124    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:10.962236 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.962209    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:10.962323 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.962236    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:10.962323 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.962261    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:10.962413 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.962394    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:10.962458 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.962441    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:10.962548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.962535    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-484zd\" (UniqueName: \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-kube-api-access-484zd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:10.962582 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:10.962560    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.063982 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.063888    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.063982 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.063945    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.063982 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.063975    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064322 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.063999    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-484zd\" (UniqueName: \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-kube-api-access-484zd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064322 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.064021    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064322 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.064074    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064322 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.064100    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064322 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.064223    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064322 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.064284    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064662 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.064455    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064662 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.064615    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064766 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.064733    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064821 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.064810    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.064974 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.064946    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.066365 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.066345    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.066478 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.066462    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.120905 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.120837    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.121028 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.120958    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-484zd\" (UniqueName: \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-kube-api-access-484zd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7zrcd\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.167620 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.167584    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:11.593539 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.593507    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"]
Oct 30 18:51:11.632620 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:51:11.632588    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bf2ab1_b51b_4d67_b722_1ff40f17eef9.slice/crio-ce227e238919a59448746e3e9c2f5d67d2864fee076148d8958db25e7f4e6969 WatchSource:0}: Error finding container ce227e238919a59448746e3e9c2f5d67d2864fee076148d8958db25e7f4e6969: Status 404 returned error can't find the container with id ce227e238919a59448746e3e9c2f5d67d2864fee076148d8958db25e7f4e6969
Oct 30 18:51:11.634749 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.634717    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:51:11.634826 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.634806    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:51:11.634863 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:11.634839    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:51:12.438162 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:12.438123    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" event={"ID":"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9","Type":"ContainerStarted","Data":"e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af"}
Oct 30 18:51:12.438561 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:12.438180    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" event={"ID":"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9","Type":"ContainerStarted","Data":"ce227e238919a59448746e3e9c2f5d67d2864fee076148d8958db25e7f4e6969"}
Oct 30 18:51:12.652378 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:12.652326    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" podStartSLOduration=2.652311011 podStartE2EDuration="2.652311011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:12.602309837 +0000 UTC m=+763.490226926" watchObservedRunningTime="2026-10-30 18:51:12.652311011 +0000 UTC m=+763.540228099"
Oct 30 18:51:13.167973 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:13.167937    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:13.169417 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:13.169382    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7zrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:13.169543 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:13.169441    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 18:51:14.168087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:14.168054    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7zrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:14.168471 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:14.168120    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 18:51:15.168053 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:15.168014    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7zrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:15.168235 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:15.168079    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 18:51:16.168409 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:16.168371    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7zrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:16.168817 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:16.168446    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 18:51:17.168619 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:17.168586    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7zrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:17.169117 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:17.168638    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 18:51:18.168639 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:18.168598    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7zrcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:51:18.169194 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:18.168674    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 18:51:19.929253 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:19.929219    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:19.929725 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:19.929616    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:19.931006 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:19.930984    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 18:51:20.157824 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:51:20.157788    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:51:50.157097 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:51:50.157055    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:51:59.423424 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.423380    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"]
Oct 30 18:51:59.426895 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.426868    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.479959 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.479928    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1290a6-openshift-default-dockercfg-zvdjm\""
Oct 30 18:51:59.531543 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.531506    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"]
Oct 30 18:51:59.551387 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.551351    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.551582 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.551390    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.551582 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.551419    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.551582 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.551534    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhtr\" (UniqueName: \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-kube-api-access-8hhtr\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.551582 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.551575    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.551722 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.551602    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.551722 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.551690    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.551783 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.551757    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.551783 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.551776    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.653234 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653194    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.653455 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653247    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.653455 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653285    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.653455 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653315    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhtr\" (UniqueName: \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-kube-api-access-8hhtr\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.653455 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653361    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.653455 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653397    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.653455 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653425    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.653836 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653558    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.653836 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653583    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.653836 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653694    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.654005 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.653932    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.654306 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.654256    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.654518 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.654467    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.654719 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.654696    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.656712 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.656684    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.656821 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.656786    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.710160 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.710126    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.710364 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.710215    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhtr\" (UniqueName: \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-kube-api-access-8hhtr\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:51:59.740108 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:51:59.740074    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:52:00.234462 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.234432    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"]
Oct 30 18:52:00.235790 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:52:00.235758    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf09e949_f7ee_4e52_8a6d_a789cf00b7d2.slice/crio-c5e9a61ab3f962f3c2d76710da5bc491d3a1eff2e05477eae1aec03cbf61e6f8 WatchSource:0}: Error finding container c5e9a61ab3f962f3c2d76710da5bc491d3a1eff2e05477eae1aec03cbf61e6f8: Status 404 returned error can't find the container with id c5e9a61ab3f962f3c2d76710da5bc491d3a1eff2e05477eae1aec03cbf61e6f8
Oct 30 18:52:00.238137 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.238102    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:00.238226 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.238180    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:00.238226 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.238217    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:00.370901 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.370865    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"]
Oct 30 18:52:00.374874 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.374848    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.427026 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.426990    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-66bdc0-openshift-default-dockercfg-d69f7\""
Oct 30 18:52:00.435803 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.435750    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"]
Oct 30 18:52:00.561772 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.561687    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.561772 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.561736    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.561959 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.561804    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.561959 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.561856    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.561959 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.561882    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87fb6889-da6f-4d12-984f-4d0a6f217f05-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.561959 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.561930    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.562108 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.561960    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.562108 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.562017    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.562108 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.562047    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96n2p\" (UniqueName: \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-kube-api-access-96n2p\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.662611 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.662573    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.662806 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.662635    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.662806 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.662667    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.662806 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.662696    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.662806 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.662726    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87fb6889-da6f-4d12-984f-4d0a6f217f05-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.662806 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.662796    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.663066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.662816    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.663066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.662848    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.663066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.662897    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96n2p\" (UniqueName: \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-kube-api-access-96n2p\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.663066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.663032    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.663333 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.663298    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" event={"ID":"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2","Type":"ContainerStarted","Data":"8bac6f81dbf151ced9883fa5f0645228bf991f6246d50725d877593efe1d0f27"}
Oct 30 18:52:00.663395 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.663327    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.663463 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.663318    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.663463 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.663434    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" event={"ID":"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2","Type":"ContainerStarted","Data":"c5e9a61ab3f962f3c2d76710da5bc491d3a1eff2e05477eae1aec03cbf61e6f8"}
Oct 30 18:52:00.663727 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.663699    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87fb6889-da6f-4d12-984f-4d0a6f217f05-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.664303 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.664279    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.665605 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.665583    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.665755 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.665733    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.723234 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.723201    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.723654 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.723632    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96n2p\" (UniqueName: \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-kube-api-access-96n2p\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:00.740379 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.740348    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:52:00.783261 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.783207    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" podStartSLOduration=1.7831923619999999 podStartE2EDuration="1.783192362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:00.782597834 +0000 UTC m=+811.670514917" watchObservedRunningTime="2026-10-30 18:52:00.783192362 +0000 UTC m=+811.671109452"
Oct 30 18:52:00.988061 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:00.988009    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:01.311637 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.311558    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"]
Oct 30 18:52:01.316250 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.316232    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.368481 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.368446    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bc98e1-openshift-default-dockercfg-mrm2g\""
Oct 30 18:52:01.374413 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.374386    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"]
Oct 30 18:52:01.471987 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.471955    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"]
Oct 30 18:52:01.472415 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.472229    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64927929-46af-4325-80de-45c8eca88e5d-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.472415 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.472280    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.472415 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.472300    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.472415 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.472411    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.472679 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.472434    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.472679 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.472468    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.472679 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.472581    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsnl\" (UniqueName: \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-kube-api-access-wgsnl\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.472679 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.472609    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64927929-46af-4325-80de-45c8eca88e5d-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.472679 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.472626    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.473791 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:52:01.473766    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fb6889_da6f_4d12_984f_4d0a6f217f05.slice/crio-2a5bb7bbd0a6059f2b9433ded8ce0fc837896b5e90755953b59071586d1b7549 WatchSource:0}: Error finding container 2a5bb7bbd0a6059f2b9433ded8ce0fc837896b5e90755953b59071586d1b7549: Status 404 returned error can't find the container with id 2a5bb7bbd0a6059f2b9433ded8ce0fc837896b5e90755953b59071586d1b7549
Oct 30 18:52:01.475914 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.475883    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:01.475988 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.475944    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:01.475988 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.475972    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:01.573967 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.573880    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64927929-46af-4325-80de-45c8eca88e5d-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.573967 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.573927    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.573967 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.573958    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.574244 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.573992    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.574244 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.574018    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.574244 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.574167    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.574398 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.574262    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsnl\" (UniqueName: \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-kube-api-access-wgsnl\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.574398 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.574300    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64927929-46af-4325-80de-45c8eca88e5d-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.574398 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.574329    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.574568 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.574467    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.574660 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.574640    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.574733 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.574664    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.574919 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.574896    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.575271 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.575251    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64927929-46af-4325-80de-45c8eca88e5d-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.576237 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.576213    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.576903 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.576878    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64927929-46af-4325-80de-45c8eca88e5d-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.631605 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.631572    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.631777 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.631710    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsnl\" (UniqueName: \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-kube-api-access-wgsnl\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.670556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.670517    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" event={"ID":"87fb6889-da6f-4d12-984f-4d0a6f217f05","Type":"ContainerStarted","Data":"3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8"}
Oct 30 18:52:01.670720 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.670578    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" event={"ID":"87fb6889-da6f-4d12-984f-4d0a6f217f05","Type":"ContainerStarted","Data":"2a5bb7bbd0a6059f2b9433ded8ce0fc837896b5e90755953b59071586d1b7549"}
Oct 30 18:52:01.745974 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.745939    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:52:01.790030 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.789968    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" podStartSLOduration=1.789953082 podStartE2EDuration="1.789953082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:01.786979093 +0000 UTC m=+812.674896195" watchObservedRunningTime="2026-10-30 18:52:01.789953082 +0000 UTC m=+812.677870172"
Oct 30 18:52:01.928914 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.928833    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:01.988970 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:01.988944    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:02.352430 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.351366    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"]
Oct 30 18:52:02.361030 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.359067    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:02.361030 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.359176    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:02.361030 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.359227    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:02.678317 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.678228    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" event={"ID":"64927929-46af-4325-80de-45c8eca88e5d","Type":"ContainerStarted","Data":"944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f"}
Oct 30 18:52:02.678809 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.678790    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:52:02.678890 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.678842    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" event={"ID":"64927929-46af-4325-80de-45c8eca88e5d","Type":"ContainerStarted","Data":"e8c4759ffd050d677e6b287035bfe78b30d8a3dfccfc1b7c466e20e0ddc1542c"}
Oct 30 18:52:02.679403 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.679387    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:52:02.795865 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.795809    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" podStartSLOduration=1.795792118 podStartE2EDuration="1.795792118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:02.79546268 +0000 UTC m=+813.683379770" watchObservedRunningTime="2026-10-30 18:52:02.795792118 +0000 UTC m=+813.683709267"
Oct 30 18:52:02.937686 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.937640    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:02.994043 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:02.994013    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:03.683831 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:03.683799    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:03.684548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:03.684525    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:03.935128 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:03.935050    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:04.692781 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:04.692750    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:04.693599 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:04.693578    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:52:13.668587 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.668548    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"]
Oct 30 18:52:13.672293 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.672267    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 18:52:13.697252 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.697222    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8dx\" (UniqueName: \"kubernetes.io/projected/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-kube-api-access-bn8dx\") pod \"payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj\" (UID: \"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 18:52:13.697517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.697462    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj\" (UID: \"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 18:52:13.724484 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.724444    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-41db7c\""
Oct 30 18:52:13.774319 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.774284    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-41db7c-dockercfg-6j457\""
Oct 30 18:52:13.776404 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.776383    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"]
Oct 30 18:52:13.798387 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.798350    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj\" (UID: \"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 18:52:13.798591 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.798397    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8dx\" (UniqueName: \"kubernetes.io/projected/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-kube-api-access-bn8dx\") pod \"payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj\" (UID: \"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 18:52:13.799107 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.799087    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj\" (UID: \"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 18:52:13.863643 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.863611    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8dx\" (UniqueName: \"kubernetes.io/projected/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-kube-api-access-bn8dx\") pod \"payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj\" (UID: \"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 18:52:13.984587 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:13.984545    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 18:52:14.359208 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:14.359182    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"]
Oct 30 18:52:14.360480 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:52:14.360451    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a5e580_a4ab_4fd7_8b9a_a92367ddeaaa.slice/crio-62d50155f3d6106b68a1fbef93c3308ec9996f6d2be304eaca8d49cbf1f856e4 WatchSource:0}: Error finding container 62d50155f3d6106b68a1fbef93c3308ec9996f6d2be304eaca8d49cbf1f856e4: Status 404 returned error can't find the container with id 62d50155f3d6106b68a1fbef93c3308ec9996f6d2be304eaca8d49cbf1f856e4
Oct 30 18:52:14.750885 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:14.750850    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj" event={"ID":"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa","Type":"ContainerStarted","Data":"62d50155f3d6106b68a1fbef93c3308ec9996f6d2be304eaca8d49cbf1f856e4"}
Oct 30 18:52:15.964399 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:15.964361    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"]
Oct 30 18:52:15.967796 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:15.967776    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 18:52:16.020656 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.020622    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b52af405-150a-4af8-9bde-a9afad214d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd\" (UID: \"b52af405-150a-4af8-9bde-a9afad214d47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 18:52:16.020838 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.020748    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7h9\" (UniqueName: \"kubernetes.io/projected/b52af405-150a-4af8-9bde-a9afad214d47-kube-api-access-5f7h9\") pod \"payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd\" (UID: \"b52af405-150a-4af8-9bde-a9afad214d47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 18:52:16.026032 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.026007    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"]
Oct 30 18:52:16.078364 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.078031    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-66bdc0-dockercfg-8hnmc\""
Oct 30 18:52:16.078364 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.078080    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-66bdc0\""
Oct 30 18:52:16.121879 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.121839    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b52af405-150a-4af8-9bde-a9afad214d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd\" (UID: \"b52af405-150a-4af8-9bde-a9afad214d47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 18:52:16.122089 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.121941    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7h9\" (UniqueName: \"kubernetes.io/projected/b52af405-150a-4af8-9bde-a9afad214d47-kube-api-access-5f7h9\") pod \"payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd\" (UID: \"b52af405-150a-4af8-9bde-a9afad214d47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 18:52:16.122681 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.122659    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b52af405-150a-4af8-9bde-a9afad214d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd\" (UID: \"b52af405-150a-4af8-9bde-a9afad214d47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 18:52:16.179703 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.179673    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7h9\" (UniqueName: \"kubernetes.io/projected/b52af405-150a-4af8-9bde-a9afad214d47-kube-api-access-5f7h9\") pod \"payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd\" (UID: \"b52af405-150a-4af8-9bde-a9afad214d47\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 18:52:16.283573 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.283469    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 18:52:16.663532 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.663502    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"]
Oct 30 18:52:16.665750 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:52:16.665719    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52af405_150a_4af8_9bde_a9afad214d47.slice/crio-c5a94913ebc1f9db4265802237d4dd5e381b1166b5b5a0b8c9235295196074aa WatchSource:0}: Error finding container c5a94913ebc1f9db4265802237d4dd5e381b1166b5b5a0b8c9235295196074aa: Status 404 returned error can't find the container with id c5a94913ebc1f9db4265802237d4dd5e381b1166b5b5a0b8c9235295196074aa
Oct 30 18:52:16.764815 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:16.764769    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd" event={"ID":"b52af405-150a-4af8-9bde-a9afad214d47","Type":"ContainerStarted","Data":"c5a94913ebc1f9db4265802237d4dd5e381b1166b5b5a0b8c9235295196074aa"}
Oct 30 18:52:17.782943 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:17.782843    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj" event={"ID":"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa","Type":"ContainerStarted","Data":"b06dee0f36c748139250a94141aa7a998af464415d329827dfa0b8fc2b6d87ee"}
Oct 30 18:52:17.783421 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:17.782988    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 18:52:17.786204 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:17.786169    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd" event={"ID":"b52af405-150a-4af8-9bde-a9afad214d47","Type":"ContainerStarted","Data":"83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4"}
Oct 30 18:52:17.786469 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:17.786451    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 18:52:17.899356 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:17.899292    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj" podStartSLOduration=2.016860557 podStartE2EDuration="4.89927856s" totalImagesPullingTime="2.882418003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:17.897538258 +0000 UTC m=+828.785455347" watchObservedRunningTime="2026-10-30 18:52:17.89927856 +0000 UTC m=+828.787195664"
Oct 30 18:52:18.013677 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:18.013466    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd" podStartSLOduration=2.222131752 podStartE2EDuration="3.013446561s" totalImagesPullingTime="791.314809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:18.012856574 +0000 UTC m=+828.900773665" watchObservedRunningTime="2026-10-30 18:52:18.013446561 +0000 UTC m=+828.901363650"
Oct 30 18:52:20.157298 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:52:20.157259    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:52:20.776678 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:20.776642    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"]
Oct 30 18:52:20.780310 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:20.780293    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 18:52:20.832684 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:20.832656    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-975cf5\""
Oct 30 18:52:20.876679 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:20.876627    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4547d45a-763d-485c-9c2f-9433013e97af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r\" (UID: \"4547d45a-763d-485c-9c2f-9433013e97af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 18:52:20.876926 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:20.876818    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p47c\" (UniqueName: \"kubernetes.io/projected/4547d45a-763d-485c-9c2f-9433013e97af-kube-api-access-2p47c\") pod \"payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r\" (UID: \"4547d45a-763d-485c-9c2f-9433013e97af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 18:52:20.882092 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:20.882063    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-975cf5-dockercfg-s4694\""
Oct 30 18:52:20.884627 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:20.884604    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"]
Oct 30 18:52:20.977958 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:20.977927    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4547d45a-763d-485c-9c2f-9433013e97af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r\" (UID: \"4547d45a-763d-485c-9c2f-9433013e97af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 18:52:20.978227 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:20.977994    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p47c\" (UniqueName: \"kubernetes.io/projected/4547d45a-763d-485c-9c2f-9433013e97af-kube-api-access-2p47c\") pod \"payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r\" (UID: \"4547d45a-763d-485c-9c2f-9433013e97af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 18:52:20.978918 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:20.978893    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4547d45a-763d-485c-9c2f-9433013e97af-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r\" (UID: \"4547d45a-763d-485c-9c2f-9433013e97af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 18:52:21.036861 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:21.036783    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p47c\" (UniqueName: \"kubernetes.io/projected/4547d45a-763d-485c-9c2f-9433013e97af-kube-api-access-2p47c\") pod \"payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r\" (UID: \"4547d45a-763d-485c-9c2f-9433013e97af\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 18:52:21.092706 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:21.092669    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 18:52:21.463388 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:21.463363    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"]
Oct 30 18:52:21.464879 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:52:21.464853    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4547d45a_763d_485c_9c2f_9433013e97af.slice/crio-43cf957842961aec9465083755729cc707b42ee11fd5a3ffacdbfee75db6aefe WatchSource:0}: Error finding container 43cf957842961aec9465083755729cc707b42ee11fd5a3ffacdbfee75db6aefe: Status 404 returned error can't find the container with id 43cf957842961aec9465083755729cc707b42ee11fd5a3ffacdbfee75db6aefe
Oct 30 18:52:21.812795 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:21.812711    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r" event={"ID":"4547d45a-763d-485c-9c2f-9433013e97af","Type":"ContainerStarted","Data":"d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160"}
Oct 30 18:52:21.812795 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:21.812764    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r" event={"ID":"4547d45a-763d-485c-9c2f-9433013e97af","Type":"ContainerStarted","Data":"43cf957842961aec9465083755729cc707b42ee11fd5a3ffacdbfee75db6aefe"}
Oct 30 18:52:21.813021 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:21.812921    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 18:52:21.927899 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:21.927843    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r" podStartSLOduration=1.927829691 podStartE2EDuration="1.927829691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:21.92575681 +0000 UTC m=+832.813673896" watchObservedRunningTime="2026-10-30 18:52:21.927829691 +0000 UTC m=+832.815746780"
Oct 30 18:52:23.434946 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.434910    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"]
Oct 30 18:52:23.438424 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.438405    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 18:52:23.493454 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.493417    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-611053\""
Oct 30 18:52:23.502948 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.502915    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96769b37-30bd-4d6c-8035-39bffcbc1f34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-611053-569b766769-qjkgj\" (UID: \"96769b37-30bd-4d6c-8035-39bffcbc1f34\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 18:52:23.503124 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.503033    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztv7\" (UniqueName: \"kubernetes.io/projected/96769b37-30bd-4d6c-8035-39bffcbc1f34-kube-api-access-cztv7\") pod \"payload-processing-e2e-worker-w5-611053-569b766769-qjkgj\" (UID: \"96769b37-30bd-4d6c-8035-39bffcbc1f34\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 18:52:23.543312 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.543282    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-611053-dockercfg-6jr7j\""
Oct 30 18:52:23.545770 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.545745    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"]
Oct 30 18:52:23.603946 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.603854    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96769b37-30bd-4d6c-8035-39bffcbc1f34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-611053-569b766769-qjkgj\" (UID: \"96769b37-30bd-4d6c-8035-39bffcbc1f34\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 18:52:23.604121 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.603951    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztv7\" (UniqueName: \"kubernetes.io/projected/96769b37-30bd-4d6c-8035-39bffcbc1f34-kube-api-access-cztv7\") pod \"payload-processing-e2e-worker-w5-611053-569b766769-qjkgj\" (UID: \"96769b37-30bd-4d6c-8035-39bffcbc1f34\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 18:52:23.604602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.604582    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96769b37-30bd-4d6c-8035-39bffcbc1f34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-611053-569b766769-qjkgj\" (UID: \"96769b37-30bd-4d6c-8035-39bffcbc1f34\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 18:52:23.660757 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.660729    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztv7\" (UniqueName: \"kubernetes.io/projected/96769b37-30bd-4d6c-8035-39bffcbc1f34-kube-api-access-cztv7\") pod \"payload-processing-e2e-worker-w5-611053-569b766769-qjkgj\" (UID: \"96769b37-30bd-4d6c-8035-39bffcbc1f34\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 18:52:23.751099 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:23.751055    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 18:52:24.133482 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:24.133452    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"]
Oct 30 18:52:24.134296 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:52:24.134267    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96769b37_30bd_4d6c_8035_39bffcbc1f34.slice/crio-67c4d5d1e68858f4bbbbd47621abfe264371e3db61f3ab24b9180efd2b2a5780 WatchSource:0}: Error finding container 67c4d5d1e68858f4bbbbd47621abfe264371e3db61f3ab24b9180efd2b2a5780: Status 404 returned error can't find the container with id 67c4d5d1e68858f4bbbbd47621abfe264371e3db61f3ab24b9180efd2b2a5780
Oct 30 18:52:24.832439 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:24.832401    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj" event={"ID":"96769b37-30bd-4d6c-8035-39bffcbc1f34","Type":"ContainerStarted","Data":"cde569a9a9062ebf30f597444247435bef27b136d9c148ed33b13c5816ceb5d7"}
Oct 30 18:52:24.832904 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:24.832452    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj" event={"ID":"96769b37-30bd-4d6c-8035-39bffcbc1f34","Type":"ContainerStarted","Data":"67c4d5d1e68858f4bbbbd47621abfe264371e3db61f3ab24b9180efd2b2a5780"}
Oct 30 18:52:24.832904 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:24.832527    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 18:52:24.946530 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:24.946459    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj" podStartSLOduration=1.946443936 podStartE2EDuration="1.946443936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:24.944979917 +0000 UTC m=+835.832897007" watchObservedRunningTime="2026-10-30 18:52:24.946443936 +0000 UTC m=+835.834361067"
Oct 30 18:52:28.795532 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:28.795476    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 18:52:28.795924 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:28.795663    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 18:52:30.382687 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.382651    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"]
Oct 30 18:52:30.386237 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.386217    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.440275 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.440160    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:52:30.440275 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.440169    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:52:30.443749 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.443719    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"]
Oct 30 18:52:30.489920 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.489885    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-1290a6-serving-cert\""
Oct 30 18:52:30.489920 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.489893    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9wflp\""
Oct 30 18:52:30.490147 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.489949    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 18:52:30.493530 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.493488    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mp9g\" (UniqueName: \"kubernetes.io/projected/ec566702-3dbf-4722-8a8b-a54f08cda9e9-kube-api-access-8mp9g\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.493609 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.493571    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec566702-3dbf-4722-8a8b-a54f08cda9e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.493718 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.493696    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.493758 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.493746    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-metrics-certs\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.594640 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.594597    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec566702-3dbf-4722-8a8b-a54f08cda9e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.594837 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.594708    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.594837 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.594746    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-metrics-certs\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.594837 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.594790    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mp9g\" (UniqueName: \"kubernetes.io/projected/ec566702-3dbf-4722-8a8b-a54f08cda9e9-kube-api-access-8mp9g\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.595006 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:52:30.594855    4267 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-serving-cert: secret "maas-api-e2e-worker-w4-1290a6-serving-cert" not found
Oct 30 18:52:30.595006 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:52:30.594920    4267 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-maas-api-tls podName:ec566702-3dbf-4722-8a8b-a54f08cda9e9 nodeName:}" failed. No retries permitted until 2026-10-30 18:52:31.094904315 +0000 UTC m=+841.982821382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-maas-api-tls") pod "maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc" (UID: "ec566702-3dbf-4722-8a8b-a54f08cda9e9") : secret "maas-api-e2e-worker-w4-1290a6-serving-cert" not found
Oct 30 18:52:30.595359 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.595336    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec566702-3dbf-4722-8a8b-a54f08cda9e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.597365 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.597345    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-metrics-certs\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.636633 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.636539    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"]
Oct 30 18:52:30.640388 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.640367    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 18:52:30.651713 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.651686    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mp9g\" (UniqueName: \"kubernetes.io/projected/ec566702-3dbf-4722-8a8b-a54f08cda9e9-kube-api-access-8mp9g\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:30.693669 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.693643    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-bc98e1\""
Oct 30 18:52:30.694024 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.693828    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-bc98e1-dockercfg-9wbvg\""
Oct 30 18:52:30.694252 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.694228    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"]
Oct 30 18:52:30.695524 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.695479    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj55d\" (UniqueName: \"kubernetes.io/projected/53707cb2-f063-4273-bd83-5dc56d5e2c77-kube-api-access-bj55d\") pod \"payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m\" (UID: \"53707cb2-f063-4273-bd83-5dc56d5e2c77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 18:52:30.695637 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.695599    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53707cb2-f063-4273-bd83-5dc56d5e2c77-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m\" (UID: \"53707cb2-f063-4273-bd83-5dc56d5e2c77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 18:52:30.797155 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.797105    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53707cb2-f063-4273-bd83-5dc56d5e2c77-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m\" (UID: \"53707cb2-f063-4273-bd83-5dc56d5e2c77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 18:52:30.799000 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.798964    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53707cb2-f063-4273-bd83-5dc56d5e2c77-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m\" (UID: \"53707cb2-f063-4273-bd83-5dc56d5e2c77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 18:52:30.799138 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.799027    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj55d\" (UniqueName: \"kubernetes.io/projected/53707cb2-f063-4273-bd83-5dc56d5e2c77-kube-api-access-bj55d\") pod \"payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m\" (UID: \"53707cb2-f063-4273-bd83-5dc56d5e2c77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 18:52:30.858054 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.858022    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj55d\" (UniqueName: \"kubernetes.io/projected/53707cb2-f063-4273-bd83-5dc56d5e2c77-kube-api-access-bj55d\") pod \"payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m\" (UID: \"53707cb2-f063-4273-bd83-5dc56d5e2c77\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 18:52:30.954771 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:30.954736    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 18:52:31.102013 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:31.101977    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:31.104346 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:31.104324    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:31.298677 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:31.298589    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:31.344379 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:31.344339    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"]
Oct 30 18:52:31.705545 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:31.705521    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"]
Oct 30 18:52:31.707155 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:52:31.707127    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec566702_3dbf_4722_8a8b_a54f08cda9e9.slice/crio-9712d67c92935de651b56aaa16271437e20ccbcc03043cd307c5b7f375ac42a8 WatchSource:0}: Error finding container 9712d67c92935de651b56aaa16271437e20ccbcc03043cd307c5b7f375ac42a8: Status 404 returned error can't find the container with id 9712d67c92935de651b56aaa16271437e20ccbcc03043cd307c5b7f375ac42a8
Oct 30 18:52:31.872080 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:31.872041    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m" event={"ID":"53707cb2-f063-4273-bd83-5dc56d5e2c77","Type":"ContainerStarted","Data":"ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190"}
Oct 30 18:52:31.872290 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:31.872133    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m" event={"ID":"53707cb2-f063-4273-bd83-5dc56d5e2c77","Type":"ContainerStarted","Data":"fefbc9100aaf34c3b6698256842a85bbe7a90e941c249d444f51d7ab8f554a83"}
Oct 30 18:52:31.872290 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:31.872191    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 18:52:31.873333 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:31.873312    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc" event={"ID":"ec566702-3dbf-4722-8a8b-a54f08cda9e9","Type":"ContainerStarted","Data":"9712d67c92935de651b56aaa16271437e20ccbcc03043cd307c5b7f375ac42a8"}
Oct 30 18:52:31.999978 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:31.999859    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m" podStartSLOduration=1.999839578 podStartE2EDuration="1.999839578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:31.99861207 +0000 UTC m=+842.886529161" watchObservedRunningTime="2026-10-30 18:52:31.999839578 +0000 UTC m=+842.887756668"
Oct 30 18:52:32.822110 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:32.822029    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 18:52:34.728709 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:34.728661    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:52:34.729249 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:34.728740    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:35.843090 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:35.843052    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 18:52:35.901162 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:35.901084    4267 generic.go:356] "Generic (PLEG): container finished" podID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerID="3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8" exitCode=0
Oct 30 18:52:35.901312 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:35.901157    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" event={"ID":"87fb6889-da6f-4d12-984f-4d0a6f217f05","Type":"ContainerDied","Data":"3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8"}
Oct 30 18:52:35.901568 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:35.901551    4267 scope.go:122] "RemoveContainer" containerID="3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8"
Oct 30 18:52:35.902819 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:35.902790    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:35.902916 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:35.902857    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:35.902916 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:35.902888    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc" event={"ID":"ec566702-3dbf-4722-8a8b-a54f08cda9e9","Type":"ContainerStarted","Data":"5dcaf96053e7a2334efba9e554b2c5614bae97fb7c90638daccdc41c99efad16"}
Oct 30 18:52:35.902916 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:35.902905    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:35.903061 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:35.902999    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:36.077586 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:36.077436    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc" podStartSLOduration=2.832767568 podStartE2EDuration="6.077420273s" totalImagesPullingTime="3.244652705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:36.074894302 +0000 UTC m=+846.962811402" watchObservedRunningTime="2026-10-30 18:52:36.077420273 +0000 UTC m=+846.965337362"
Oct 30 18:52:36.927516 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:36.923572    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" event={"ID":"87fb6889-da6f-4d12-984f-4d0a6f217f05","Type":"ContainerStarted","Data":"53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9"}
Oct 30 18:52:36.989209 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:36.989174    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:36.989484 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:36.989460    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 18:52:36.989574 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:36.989553    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 18:52:37.989366 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:37.989326    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 18:52:37.989779 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:37.989393    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 18:52:38.989085 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:38.989053    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 18:52:38.989292 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:38.989103    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 18:52:39.989419 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:39.989387    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 18:52:39.989835 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:39.989445    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 18:52:40.988773 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:40.988688    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 18:52:40.988773 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:40.988760    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 18:52:41.945977 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:41.945940    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 18:52:41.988945 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:41.988908    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 18:52:41.989111 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:41.988974    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 18:52:42.885049 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:42.885023    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 18:52:42.993056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:42.993023    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:42.993426 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:42.993307    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:42.994040 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:42.994021    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 18:52:47.912131 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:47.912092    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"]
Oct 30 18:52:47.914684 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:47.914662    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:47.967190 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:47.967154    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-41db7c\"/\"openshift-service-ca.crt\""
Oct 30 18:52:47.974548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:47.974466    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tmp-dir\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:47.974714 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:47.974611    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-home\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:47.974714 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:47.974656    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d74j\" (UniqueName: \"kubernetes.io/projected/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-kube-api-access-7d74j\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:47.975649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:47.974915    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-dshm\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:47.975649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:47.975018    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tls-certs\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:47.975649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:47.975068    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-model-cache\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.016567 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.016517    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-41db7c\"/\"kube-root-ca.crt\""
Oct 30 18:52:48.016755 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.016662    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-41db7c\"/\"default-dockercfg-46k5f\""
Oct 30 18:52:48.016755 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.016683    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-41db7c\"/\"simulator-w6-41db7c-kserve-self-signed-certs\""
Oct 30 18:52:48.019167 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.019146    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"]
Oct 30 18:52:48.075945 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.075910    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-home\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.076139 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.075953    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d74j\" (UniqueName: \"kubernetes.io/projected/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-kube-api-access-7d74j\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.076139 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.076035    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-dshm\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.076139 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.076091    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tls-certs\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.076311 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.076146    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-model-cache\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.076311 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.076175    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tmp-dir\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.076413 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.076360    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-home\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.076605 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.076586    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tmp-dir\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.076702 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.076656    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-model-cache\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.079787 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.079762    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tls-certs\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.080162 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.080132    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-dshm\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.132726 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.132689    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d74j\" (UniqueName: \"kubernetes.io/projected/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-kube-api-access-7d74j\") pod \"simulator-w6-41db7c-kserve-7cb895548d-k24kc\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") " pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.225636 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.225598    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:48.605610 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.603289    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"]
Oct 30 18:52:48.609982 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:52:48.609949    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60ba4ae1_8289_4d1a_8e25_88c5fc57494b.slice/crio-bc6fabd1c91414156ead8ae00b6357f18f536388a6a295d1d6ec26b73807958f WatchSource:0}: Error finding container bc6fabd1c91414156ead8ae00b6357f18f536388a6a295d1d6ec26b73807958f: Status 404 returned error can't find the container with id bc6fabd1c91414156ead8ae00b6357f18f536388a6a295d1d6ec26b73807958f
Oct 30 18:52:48.721682 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.721634    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:52:48.721869 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:48.721700    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:49.011536 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:49.011473    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" event={"ID":"60ba4ae1-8289-4d1a-8e25-88c5fc57494b","Type":"ContainerStarted","Data":"c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9"}
Oct 30 18:52:49.011999 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:49.011545    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" event={"ID":"60ba4ae1-8289-4d1a-8e25-88c5fc57494b","Type":"ContainerStarted","Data":"bc6fabd1c91414156ead8ae00b6357f18f536388a6a295d1d6ec26b73807958f"}
Oct 30 18:52:49.128119 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:49.128051    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podStartSLOduration=2.128031072 podStartE2EDuration="2.128031072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:49.124397948 +0000 UTC m=+860.012315039" watchObservedRunningTime="2026-10-30 18:52:49.128031072 +0000 UTC m=+860.015948162"
Oct 30 18:52:49.227515 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:49.227465    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:50.018406 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:50.018374    4267 generic.go:356] "Generic (PLEG): container finished" podID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerID="8bac6f81dbf151ced9883fa5f0645228bf991f6246d50725d877593efe1d0f27" exitCode=0
Oct 30 18:52:50.018929 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:50.018453    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" event={"ID":"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2","Type":"ContainerDied","Data":"8bac6f81dbf151ced9883fa5f0645228bf991f6246d50725d877593efe1d0f27"}
Oct 30 18:52:50.018996 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:50.018949    4267 scope.go:122] "RemoveContainer" containerID="8bac6f81dbf151ced9883fa5f0645228bf991f6246d50725d877593efe1d0f27"
Oct 30 18:52:50.020226 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:50.020187    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:50.020347 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:50.020272    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:50.020347 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:50.020319    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:50.161038 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:52:50.160988    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:52:51.042312 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:51.042268    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" event={"ID":"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2","Type":"ContainerStarted","Data":"aca9ce4380bab96d9ef5d070ef64c55a1aaa0d6759f91c9ff31e77b7400d857c"}
Oct 30 18:52:51.740898 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:51.740859    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:52:51.741215 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:51.741193    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:52:51.741262 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:51.741240    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 18:52:52.741020 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:52.740984    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:52:52.741544 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:52.741039    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 18:52:53.741109 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:53.741066    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:52:53.741551 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:53.741129    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 18:52:54.741164 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:54.741125    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:52:54.741612 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:54.741190    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 18:52:55.741550 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:55.741509    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:52:55.741953 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:55.741568    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 18:52:56.744197 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:56.744165    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:52:56.744618 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:56.744476    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:52:56.745113 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:56.745093    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 18:52:58.225907 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:58.225873    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:58.239020 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:58.238995    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:52:59.099070 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:52:59.099042    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 18:53:08.799603 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.799571    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"]
Oct 30 18:53:08.803317 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.803293    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:08.907968 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.907936    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-66bdc0\"/\"kube-root-ca.crt\""
Oct 30 18:53:08.929087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.928840    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-66bdc0\"/\"openshift-service-ca.crt\""
Oct 30 18:53:08.929087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.928931    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66bdc0\"/\"simulator-w2-66bdc0-kserve-self-signed-certs\""
Oct 30 18:53:08.929087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.929038    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66bdc0\"/\"default-dockercfg-qkspm\""
Oct 30 18:53:08.934084 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.934057    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"]
Oct 30 18:53:08.948680 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.946201    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-dshm\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:08.948680 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.947093    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-tmp-dir\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:08.948680 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.947150    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-model-cache\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:08.948680 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.947244    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-home\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:08.948680 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.947332    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a511f699-9b06-4893-a9f7-acb661ceb6c0-tls-certs\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:08.948680 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:08.947388    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hggq6\" (UniqueName: \"kubernetes.io/projected/a511f699-9b06-4893-a9f7-acb661ceb6c0-kube-api-access-hggq6\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.048312 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.048274    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggq6\" (UniqueName: \"kubernetes.io/projected/a511f699-9b06-4893-a9f7-acb661ceb6c0-kube-api-access-hggq6\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.048507 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.048346    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-dshm\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.048507 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.048374    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-tmp-dir\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.048507 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.048392    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-model-cache\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.048507 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.048429    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-home\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.048507 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.048482    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a511f699-9b06-4893-a9f7-acb661ceb6c0-tls-certs\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.048898 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.048863    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-tmp-dir\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.049038 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.048914    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-model-cache\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.049038 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.048993    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-home\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.050722 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.050674    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-dshm\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.052183 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.052165    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a511f699-9b06-4893-a9f7-acb661ceb6c0-tls-certs\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.105066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.105036    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggq6\" (UniqueName: \"kubernetes.io/projected/a511f699-9b06-4893-a9f7-acb661ceb6c0-kube-api-access-hggq6\") pod \"simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") " pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.146594 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.146561    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:09.533518 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:09.532764    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"]
Oct 30 18:53:09.539386 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:53:09.539361    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda511f699_9b06_4893_a9f7_acb661ceb6c0.slice/crio-f1b984557a69b546ce4f4c6f212349e34644b5a4ae58dd164c8114406f2a36a3 WatchSource:0}: Error finding container f1b984557a69b546ce4f4c6f212349e34644b5a4ae58dd164c8114406f2a36a3: Status 404 returned error can't find the container with id f1b984557a69b546ce4f4c6f212349e34644b5a4ae58dd164c8114406f2a36a3
Oct 30 18:53:10.160085 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:10.160046    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" event={"ID":"a511f699-9b06-4893-a9f7-acb661ceb6c0","Type":"ContainerStarted","Data":"8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f"}
Oct 30 18:53:10.160564 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:10.160096    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" event={"ID":"a511f699-9b06-4893-a9f7-acb661ceb6c0","Type":"ContainerStarted","Data":"f1b984557a69b546ce4f4c6f212349e34644b5a4ae58dd164c8114406f2a36a3"}
Oct 30 18:53:10.284798 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:10.284733    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podStartSLOduration=2.284714765 podStartE2EDuration="2.284714765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:10.282178008 +0000 UTC m=+881.170095098" watchObservedRunningTime="2026-10-30 18:53:10.284714765 +0000 UTC m=+881.172631856"
Oct 30 18:53:11.147129 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:11.147092    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:15.195322 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:15.194620    4267 generic.go:356] "Generic (PLEG): container finished" podID="64927929-46af-4325-80de-45c8eca88e5d" containerID="944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f" exitCode=0
Oct 30 18:53:15.195322 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:15.194805    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" event={"ID":"64927929-46af-4325-80de-45c8eca88e5d","Type":"ContainerDied","Data":"944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f"}
Oct 30 18:53:15.195322 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:15.195260    4267 scope.go:122] "RemoveContainer" containerID="944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f"
Oct 30 18:53:15.198129 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:15.198099    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:15.198262 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:15.198164    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:15.198262 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:15.198198    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:16.201199 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:16.201167    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" event={"ID":"64927929-46af-4325-80de-45c8eca88e5d","Type":"ContainerStarted","Data":"1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3"}
Oct 30 18:53:16.935516 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:16.932781    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:53:16.937570 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:16.937038    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 18:53:16.937570 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:16.937094    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 18:53:17.929930 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:17.929887    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 18:53:17.930601 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:17.930558    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 18:53:18.931517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:18.930896    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 18:53:18.931517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:18.930968    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 18:53:19.147946 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:19.147907    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:19.163527 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:19.162090    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:19.257546 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:19.257512    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 18:53:19.692911 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:19.692874    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:53:19.933972 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:19.933935    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 18:53:19.934433 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:19.934020    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 18:53:20.160228 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:53:20.160148    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:53:20.929935 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:20.929894    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 18:53:20.930286 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:20.930253    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 18:53:21.934549 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:21.934515    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:53:21.935598 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:21.935579    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 18:53:30.552718 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.552682    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"]
Oct 30 18:53:30.555653 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.555619    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.590540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.590480    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.590657 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.590585    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbb8\" (UniqueName: \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-kube-api-access-ghbb8\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.590657 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.590615    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.590728 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.590677    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae2f972-98d9-449c-9934-159b2b027f12-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.590759 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.590748    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.590827 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.590812    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.590862 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.590846    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.590896 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.590864    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.590946 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.590922    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae2f972-98d9-449c-9934-159b2b027f12-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.615660 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.615638    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-41db7c-openshift-default-dockercfg-fhcxz\""
Oct 30 18:53:30.667954 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.667921    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"]
Oct 30 18:53:30.691904 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.691872    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692091 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.691912    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae2f972-98d9-449c-9934-159b2b027f12-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692091 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.691938    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692091 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692048    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbb8\" (UniqueName: \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-kube-api-access-ghbb8\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692091 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692085    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692331 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692129    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae2f972-98d9-449c-9934-159b2b027f12-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692331 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692229    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692331 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692288    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692331 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692300    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692567 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692367    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692704 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692655    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692704 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692670    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692860 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692786    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.692896 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.692860    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae2f972-98d9-449c-9934-159b2b027f12-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.694465 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.694434    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.694686 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.694670    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae2f972-98d9-449c-9934-159b2b027f12-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.751108 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.751080    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.751225 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.751159    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbb8\" (UniqueName: \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-kube-api-access-ghbb8\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:30.867396 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:30.867311    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:31.296923 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:31.296890    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"]
Oct 30 18:53:31.297960 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:53:31.297936    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae2f972_98d9_449c_9934_159b2b027f12.slice/crio-2ea9241b7d141e51fddbea11e9617519796342d54d836e4d5a69a3da54d646ef WatchSource:0}: Error finding container 2ea9241b7d141e51fddbea11e9617519796342d54d836e4d5a69a3da54d646ef: Status 404 returned error can't find the container with id 2ea9241b7d141e51fddbea11e9617519796342d54d836e4d5a69a3da54d646ef
Oct 30 18:53:31.300415 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:31.300375    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:31.300556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:31.300463    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:31.300556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:31.300528    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:31.317420 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:31.317392    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" event={"ID":"eae2f972-98d9-449c-9934-159b2b027f12","Type":"ContainerStarted","Data":"2ea9241b7d141e51fddbea11e9617519796342d54d836e4d5a69a3da54d646ef"}
Oct 30 18:53:32.339753 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:32.339715    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" event={"ID":"eae2f972-98d9-449c-9934-159b2b027f12","Type":"ContainerStarted","Data":"d7bbcf398118f6b0c8782dc04941e9c67ea3ef486edc81e7b40cd7ab1943de36"}
Oct 30 18:53:32.868262 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:32.868225    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:32.869834 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:32.869811    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 18:53:32.869889 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:32.869861    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 18:53:33.868685 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:33.868649    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 18:53:33.869092 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:33.868707    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 18:53:34.868146 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:34.868108    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 18:53:34.868331 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:34.868178    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 18:53:35.868106 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:35.868067    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 18:53:35.868507 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:35.868127    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 18:53:36.868130 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:36.868090    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 18:53:36.868533 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:36.868152    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 18:53:37.896511 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:37.896449    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:37.897047 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:37.897019    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:37.897951 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:37.897914    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 18:53:38.067862 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:38.067808    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" podStartSLOduration=8.067793975 podStartE2EDuration="8.067793975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:32.530804776 +0000 UTC m=+903.418721867" watchObservedRunningTime="2026-10-30 18:53:38.067793975 +0000 UTC m=+908.955711063"
Oct 30 18:53:40.628213 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.628171    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"]
Oct 30 18:53:40.630764 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.630747    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.732578 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.732544    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-41db7c\"/\"premium-w6-41db7c-kserve-self-signed-certs\""
Oct 30 18:53:40.735198 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.735168    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tmp-dir\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.735324 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.735214    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tls-certs\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.735430 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.735404    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"]
Oct 30 18:53:40.735476 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.735401    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-dshm\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.735547 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.735532    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzrmv\" (UniqueName: \"kubernetes.io/projected/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-kube-api-access-fzrmv\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.735583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.735559    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-model-cache\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.735583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.735577    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-home\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.836649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.836601    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzrmv\" (UniqueName: \"kubernetes.io/projected/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-kube-api-access-fzrmv\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.836649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.836643    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-model-cache\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.836910 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.836661    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-home\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.836910 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.836696    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tmp-dir\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.836910 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.836721    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tls-certs\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.836910 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.836851    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-dshm\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.837136 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.837065    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-model-cache\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.837197 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.837143    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-home\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.837197 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.837180    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tmp-dir\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.839162 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.839140    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-dshm\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.839450 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.839431    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tls-certs\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.898464 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.898390    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzrmv\" (UniqueName: \"kubernetes.io/projected/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-kube-api-access-fzrmv\") pod \"premium-w6-41db7c-kserve-56b677d849-2rr4g\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") " pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:40.941579 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:40.941546    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:41.325393 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:41.325364    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"]
Oct 30 18:53:41.326658 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:53:41.326632    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b3b069b_5a17_4b68_afcb_4c444c3e3f0e.slice/crio-8cecca216306b5085a855f815c34fb4c2b0b35b8708d4bd1e28cfcb9b053c0b9 WatchSource:0}: Error finding container 8cecca216306b5085a855f815c34fb4c2b0b35b8708d4bd1e28cfcb9b053c0b9: Status 404 returned error can't find the container with id 8cecca216306b5085a855f815c34fb4c2b0b35b8708d4bd1e28cfcb9b053c0b9
Oct 30 18:53:41.411809 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:41.411766    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" event={"ID":"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e","Type":"ContainerStarted","Data":"8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d"}
Oct 30 18:53:41.411977 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:41.411825    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" event={"ID":"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e","Type":"ContainerStarted","Data":"8cecca216306b5085a855f815c34fb4c2b0b35b8708d4bd1e28cfcb9b053c0b9"}
Oct 30 18:53:41.538010 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:41.537957    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podStartSLOduration=1.537943673 podStartE2EDuration="1.537943673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:41.537314795 +0000 UTC m=+912.425231884" watchObservedRunningTime="2026-10-30 18:53:41.537943673 +0000 UTC m=+912.425860761"
Oct 30 18:53:41.941683 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:41.941643    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:50.177922 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:53:50.177812    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:53:50.942111 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:50.942055    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:50.959675 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:50.959648    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:53:51.487823 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:53:51.487789    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 18:54:10.673526 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.673480    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"]
Oct 30 18:54:10.676638 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.676617    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.731207 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.731172    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f5763a\"/\"openshift-service-ca.crt\""
Oct 30 18:54:10.779509 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.779450    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f5763a\"/\"default-dockercfg-64n6t\""
Oct 30 18:54:10.779509 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.779514    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f5763a\"/\"simulator-w0-f5763a-kserve-self-signed-certs\""
Oct 30 18:54:10.779733 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.779516    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f5763a\"/\"kube-root-ca.crt\""
Oct 30 18:54:10.783658 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.783633    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"]
Oct 30 18:54:10.784409 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.784387    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-dshm\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.784453 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.784420    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e634b6c-3021-428c-b87d-08912abb84f3-tls-certs\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.784574 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.784554    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thlgw\" (UniqueName: \"kubernetes.io/projected/7e634b6c-3021-428c-b87d-08912abb84f3-kube-api-access-thlgw\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.784684 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.784596    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-model-cache\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.784684 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.784644    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-tmp-dir\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.784830 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.784800    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-home\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.885840 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.885801    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-tmp-dir\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.886056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.885869    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-home\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.886056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.885911    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-dshm\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.886056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.885926    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e634b6c-3021-428c-b87d-08912abb84f3-tls-certs\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.886056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.885981    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thlgw\" (UniqueName: \"kubernetes.io/projected/7e634b6c-3021-428c-b87d-08912abb84f3-kube-api-access-thlgw\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.886056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.886008    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-model-cache\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.886672 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.886306    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-tmp-dir\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.886672 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.886345    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-home\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.886765 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.886682    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-model-cache\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.888336 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.888313    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-dshm\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.888753 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.888731    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e634b6c-3021-428c-b87d-08912abb84f3-tls-certs\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.953322 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.953288    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thlgw\" (UniqueName: \"kubernetes.io/projected/7e634b6c-3021-428c-b87d-08912abb84f3-kube-api-access-thlgw\") pod \"simulator-w0-f5763a-kserve-5f774769f-mc7jq\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") " pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:10.987704 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:10.987627    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:11.369284 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:11.369252    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"]
Oct 30 18:54:11.369519 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:54:11.369478    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e634b6c_3021_428c_b87d_08912abb84f3.slice/crio-8aa1eb13a463d5c128d240081082f48874436d1f5419bf96b9d9f410560835a1 WatchSource:0}: Error finding container 8aa1eb13a463d5c128d240081082f48874436d1f5419bf96b9d9f410560835a1: Status 404 returned error can't find the container with id 8aa1eb13a463d5c128d240081082f48874436d1f5419bf96b9d9f410560835a1
Oct 30 18:54:11.605955 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:11.605910    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" event={"ID":"7e634b6c-3021-428c-b87d-08912abb84f3","Type":"ContainerStarted","Data":"a1d20e639c9def53a34dff84c999ee7eec39ff9a16bee5c325241ad36152292e"}
Oct 30 18:54:11.606130 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:11.605965    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" event={"ID":"7e634b6c-3021-428c-b87d-08912abb84f3","Type":"ContainerStarted","Data":"8aa1eb13a463d5c128d240081082f48874436d1f5419bf96b9d9f410560835a1"}
Oct 30 18:54:11.724699 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:11.724636    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podStartSLOduration=1.724621341 podStartE2EDuration="1.724621341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:11.722975444 +0000 UTC m=+942.610892535" watchObservedRunningTime="2026-10-30 18:54:11.724621341 +0000 UTC m=+942.612538452"
Oct 30 18:54:11.988394 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:11.988308    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:20.159137 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:54:20.159095    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:54:20.987924 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:20.987882    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:21.000566 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:21.000535    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:21.672295 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:21.672266    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 18:54:44.359020 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.358989    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"]
Oct 30 18:54:44.367590 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.362235    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.435366 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.435337    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-975cf5\"/\"kube-root-ca.crt\""
Oct 30 18:54:44.467281 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.467251    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-975cf5\"/\"openshift-service-ca.crt\""
Oct 30 18:54:44.467473 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.467322    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-975cf5\"/\"default-dockercfg-5h568\""
Oct 30 18:54:44.467606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.467593    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-975cf5\"/\"premium-w1-975cf5-kserve-self-signed-certs\""
Oct 30 18:54:44.470043 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.470026    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"]
Oct 30 18:54:44.487553 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.487531    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-model-cache\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.487662 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.487571    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32606f69-4aba-496f-be5f-8070f31440d2-tls-certs\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.487718 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.487658    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-dshm\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.487758 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.487739    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-home\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.487849 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.487805    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntbxj\" (UniqueName: \"kubernetes.io/projected/32606f69-4aba-496f-be5f-8070f31440d2-kube-api-access-ntbxj\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.487894 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.487864    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-tmp-dir\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.589389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.589353    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-home\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.589586 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.589433    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntbxj\" (UniqueName: \"kubernetes.io/projected/32606f69-4aba-496f-be5f-8070f31440d2-kube-api-access-ntbxj\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.589586 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.589462    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-tmp-dir\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.589586 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.589559    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-model-cache\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.589711 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.589591    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32606f69-4aba-496f-be5f-8070f31440d2-tls-certs\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.589711 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.589629    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-dshm\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.589853 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.589827    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-home\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.589922 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.589902    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-tmp-dir\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.589972 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.589917    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-model-cache\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.591829 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.591808    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-dshm\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.592071 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.592054    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32606f69-4aba-496f-be5f-8070f31440d2-tls-certs\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.646407 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.646346    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntbxj\" (UniqueName: \"kubernetes.io/projected/32606f69-4aba-496f-be5f-8070f31440d2-kube-api-access-ntbxj\") pod \"premium-w1-975cf5-kserve-76c95f7454-98ls2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") " pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:44.674510 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:44.674469    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:45.093594 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:54:45.093555    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32606f69_4aba_496f_be5f_8070f31440d2.slice/crio-d023d28764a5229e1c81e260f79d92322f36c536b5bd346f43a8ecc98ebeda71 WatchSource:0}: Error finding container d023d28764a5229e1c81e260f79d92322f36c536b5bd346f43a8ecc98ebeda71: Status 404 returned error can't find the container with id d023d28764a5229e1c81e260f79d92322f36c536b5bd346f43a8ecc98ebeda71
Oct 30 18:54:45.094145 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:45.094122    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"]
Oct 30 18:54:45.809207 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:45.809171    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" event={"ID":"32606f69-4aba-496f-be5f-8070f31440d2","Type":"ContainerStarted","Data":"021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f"}
Oct 30 18:54:45.809707 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:45.809221    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" event={"ID":"32606f69-4aba-496f-be5f-8070f31440d2","Type":"ContainerStarted","Data":"d023d28764a5229e1c81e260f79d92322f36c536b5bd346f43a8ecc98ebeda71"}
Oct 30 18:54:45.926711 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:45.926660    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podStartSLOduration=1.926646125 podStartE2EDuration="1.926646125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:45.925032765 +0000 UTC m=+976.812949879" watchObservedRunningTime="2026-10-30 18:54:45.926646125 +0000 UTC m=+976.814563214"
Oct 30 18:54:46.675155 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:46.675123    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:50.171576 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:54:50.171533    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:54:54.675276 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:54.675237    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:54.688751 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:54.688720    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:54:54.873099 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:54:54.873065    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 18:55:20.172531 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:55:20.172470    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:55:25.680793 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.680760    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"]
Oct 30 18:55:25.684427 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.684409    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.738882 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.738854    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-611053\"/\"openshift-service-ca.crt\""
Oct 30 18:55:25.788193 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.788157    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-611053\"/\"kube-root-ca.crt\""
Oct 30 18:55:25.788589 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.788571    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-611053\"/\"premium-w5-611053-kserve-self-signed-certs\""
Oct 30 18:55:25.788715 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.788593    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-611053\"/\"default-dockercfg-h72q7\""
Oct 30 18:55:25.792745 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.792720    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"]
Oct 30 18:55:25.833315 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.833254    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b3f14f-34ea-4635-9e93-d500debd10d9-tls-certs\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.833612 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.833387    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-home\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.833612 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.833429    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w95f\" (UniqueName: \"kubernetes.io/projected/02b3f14f-34ea-4635-9e93-d500debd10d9-kube-api-access-2w95f\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.833612 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.833465    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-dshm\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.833612 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.833591    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-model-cache\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.833866 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.833634    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-tmp-dir\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.934287 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.934258    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b3f14f-34ea-4635-9e93-d500debd10d9-tls-certs\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.934486 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.934434    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-home\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.934583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.934486    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w95f\" (UniqueName: \"kubernetes.io/projected/02b3f14f-34ea-4635-9e93-d500debd10d9-kube-api-access-2w95f\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.934583 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.934562    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-dshm\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.934734 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.934669    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-model-cache\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.934734 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.934716    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-tmp-dir\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.934879 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.934843    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-home\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.935041 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.935007    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-model-cache\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.935123 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.935072    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-tmp-dir\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.936675 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.936655    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-dshm\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.936975 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.936956    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b3f14f-34ea-4635-9e93-d500debd10d9-tls-certs\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.991317 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.991283    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w95f\" (UniqueName: \"kubernetes.io/projected/02b3f14f-34ea-4635-9e93-d500debd10d9-kube-api-access-2w95f\") pod \"premium-w5-611053-kserve-849cbd585-hnb5q\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") " pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:25.997063 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:25.997042    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:26.372903 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:26.372873    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"]
Oct 30 18:55:26.375069 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:55:26.375043    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02b3f14f_34ea_4635_9e93_d500debd10d9.slice/crio-34102637eb0eb5a6044a347e39ed25356deec80d4d8a53f66bb93f09f2af262b WatchSource:0}: Error finding container 34102637eb0eb5a6044a347e39ed25356deec80d4d8a53f66bb93f09f2af262b: Status 404 returned error can't find the container with id 34102637eb0eb5a6044a347e39ed25356deec80d4d8a53f66bb93f09f2af262b
Oct 30 18:55:27.055385 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:27.055349    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" event={"ID":"02b3f14f-34ea-4635-9e93-d500debd10d9","Type":"ContainerStarted","Data":"4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810"}
Oct 30 18:55:27.055832 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:27.055401    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" event={"ID":"02b3f14f-34ea-4635-9e93-d500debd10d9","Type":"ContainerStarted","Data":"34102637eb0eb5a6044a347e39ed25356deec80d4d8a53f66bb93f09f2af262b"}
Oct 30 18:55:27.171782 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:27.171717    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podStartSLOduration=2.171698078 podStartE2EDuration="2.171698078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:27.170444197 +0000 UTC m=+1018.058361286" watchObservedRunningTime="2026-10-30 18:55:27.171698078 +0000 UTC m=+1018.059615169"
Oct 30 18:55:27.997561 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:27.997533    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:28.938134 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:28.938097    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"]
Oct 30 18:55:28.942468 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:28.942449    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:28.997137 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:28.997110    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"]
Oct 30 18:55:29.043339 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.043310    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:29.043517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.043387    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:55:29.069238 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.069211    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b6b7dad-nvglb\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:29.069383 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.069253    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b6b7dad-nvglb\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:29.069383 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.069278    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq8ww\" (UniqueName: \"kubernetes.io/projected/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-kube-api-access-xq8ww\") pod \"maas-api-revoke-sub-7b6b7dad-nvglb\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:29.170548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.170516    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b6b7dad-nvglb\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:29.170735 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.170578    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b6b7dad-nvglb\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:29.170735 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.170603    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq8ww\" (UniqueName: \"kubernetes.io/projected/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-kube-api-access-xq8ww\") pod \"maas-api-revoke-sub-7b6b7dad-nvglb\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:29.171288 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.171261    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b6b7dad-nvglb\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:29.173081 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.173062    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b6b7dad-nvglb\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:29.227713 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.227642    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq8ww\" (UniqueName: \"kubernetes.io/projected/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-kube-api-access-xq8ww\") pod \"maas-api-revoke-sub-7b6b7dad-nvglb\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:29.307650 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.307624    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:55:29.316422 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.316397    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:29.696962 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.696924    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"]
Oct 30 18:55:29.697919 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:55:29.697894    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7638e1_be67_4e71_ab34_32b05fb4c8dd.slice/crio-ff02476ac1a6152671e3cc63a9fc3e74440a018eb2ab88d2d72d2e40bcd36b06 WatchSource:0}: Error finding container ff02476ac1a6152671e3cc63a9fc3e74440a018eb2ab88d2d72d2e40bcd36b06: Status 404 returned error can't find the container with id ff02476ac1a6152671e3cc63a9fc3e74440a018eb2ab88d2d72d2e40bcd36b06
Oct 30 18:55:29.700357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:29.700336    4267 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:55:30.072775 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:30.072678    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb" event={"ID":"3d7638e1-be67-4e71-ab34-32b05fb4c8dd","Type":"ContainerStarted","Data":"ff02476ac1a6152671e3cc63a9fc3e74440a018eb2ab88d2d72d2e40bcd36b06"}
Oct 30 18:55:35.113898 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:35.113862    4267 generic.go:356] "Generic (PLEG): container finished" podID="3d7638e1-be67-4e71-ab34-32b05fb4c8dd" containerID="781390455e6125d0b933a8530d9c13fb3064251cbd564075d355093b5568d4ef" exitCode=0
Oct 30 18:55:35.114270 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:35.113952    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb" event={"ID":"3d7638e1-be67-4e71-ab34-32b05fb4c8dd","Type":"ContainerDied","Data":"781390455e6125d0b933a8530d9c13fb3064251cbd564075d355093b5568d4ef"}
Oct 30 18:55:35.998107 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:35.998071    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:36.010704 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.010674    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:36.142450 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.142421    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 18:55:36.308416 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.308391    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:36.367887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.367841    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-openshift-ca\") pod \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") "
Oct 30 18:55:36.368087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.367951    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-kube-api-access-xq8ww\" (UniqueName: \"kubernetes.io/projected/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-kube-api-access-xq8ww\") pod \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") "
Oct 30 18:55:36.368087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.368030    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-xks-ca\") pod \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\" (UID: \"3d7638e1-be67-4e71-ab34-32b05fb4c8dd\") "
Oct 30 18:55:36.368752 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.368724    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-openshift-ca" pod "3d7638e1-be67-4e71-ab34-32b05fb4c8dd" (UID: "3d7638e1-be67-4e71-ab34-32b05fb4c8dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:36.370703 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.370674    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-xks-ca" pod "3d7638e1-be67-4e71-ab34-32b05fb4c8dd" (UID: "3d7638e1-be67-4e71-ab34-32b05fb4c8dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:36.370803 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.370772    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-kube-api-access-xq8ww" pod "3d7638e1-be67-4e71-ab34-32b05fb4c8dd" (UID: "3d7638e1-be67-4e71-ab34-32b05fb4c8dd"). InnerVolumeSpecName "kube-api-access-xq8ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:36.470065 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.470028    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.470065 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.470059    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:36.470065 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:36.470069    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq8ww\" (UniqueName: \"kubernetes.io/projected/3d7638e1-be67-4e71-ab34-32b05fb4c8dd-kube-api-access-xq8ww\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:55:37.135914 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:37.135882    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"
Oct 30 18:55:37.135914 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:37.135904    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb" event={"ID":"3d7638e1-be67-4e71-ab34-32b05fb4c8dd","Type":"ContainerDied","Data":"ff02476ac1a6152671e3cc63a9fc3e74440a018eb2ab88d2d72d2e40bcd36b06"}
Oct 30 18:55:37.136107 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:37.135960    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff02476ac1a6152671e3cc63a9fc3e74440a018eb2ab88d2d72d2e40bcd36b06"
Oct 30 18:55:47.376548 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.376461    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"]
Oct 30 18:55:47.377046 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.376963    4267 cpu_manager.go:414] "containerMap: removing container" podUID="3d7638e1-be67-4e71-ab34-32b05fb4c8dd" containerName="revoke-keys"
Oct 30 18:55:47.377046 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.376976    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7638e1-be67-4e71-ab34-32b05fb4c8dd" containerName="revoke-keys"
Oct 30 18:55:47.377046 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.377047    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7638e1-be67-4e71-ab34-32b05fb4c8dd" containerName="revoke-keys"
Oct 30 18:55:47.380113 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.380093    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.481450 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.481410    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"]
Oct 30 18:55:47.481754 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.481607    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.481865 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.481782    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjl2w\" (UniqueName: \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-kube-api-access-vjl2w\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.481865 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.481840    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.481945 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.481880    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.481988 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.481942    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.482021 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.482012    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.482111 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.482094    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.482166 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.482152    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.482202 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.482190    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.583277 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583241    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.583488 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583310    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.583488 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583351    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.583488 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583374    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.583488 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583409    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.583488 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583442    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjl2w\" (UniqueName: \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-kube-api-access-vjl2w\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.583790 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583503    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.583790 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583536    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.583790 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583596    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.583790 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583759    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.584003 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583784    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.584003 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.583933    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.584145 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.584126    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.584586 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.584556    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.585947 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.585923    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.586463 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.586443    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.641771 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.641697    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjl2w\" (UniqueName: \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-kube-api-access-vjl2w\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.641953 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.641927    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.693096 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.693055    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:47.874816 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.874776    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"]
Oct 30 18:55:47.878747 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.878724    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:47.936292 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.936255    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-611053-openshift-default-dockercfg-7h2nn\""
Oct 30 18:55:47.940449 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.940416    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"]
Oct 30 18:55:47.989538 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.988618    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:47.989881 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.989852    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:47.990033 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.990018    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:47.990196 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.990161    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:47.990315 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.990297    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:47.990453 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.990420    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:47.990707 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.990687    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:47.991025 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.991008    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:47.991194 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:47.991175    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4xcz\" (UniqueName: \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-kube-api-access-n4xcz\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.092702 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.092661    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.093427 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.093399    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.094416 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.094398    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.094949 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.094932    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.095358 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.095342    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4xcz\" (UniqueName: \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-kube-api-access-n4xcz\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.096048 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.096026    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.096137 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.096076    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.096137 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.096115    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.096246 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.096144    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.096602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.094257    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.096734 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.095292    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.096836 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.094810    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.096927 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.096825    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.097024 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.095947    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.097125 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.097079    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.099188 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.099166    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.151721 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.151686    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4xcz\" (UniqueName: \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-kube-api-access-n4xcz\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.151880 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.151741    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.160158 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.160127    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"]
Oct 30 18:55:48.162315 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:55:48.162286    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3a967e_d63e_46e5_ba22_292d2c997aa6.slice/crio-288c5427ebaab80fc1d1fe45bd10c44557dff114eed209f22cf64d45f927138d WatchSource:0}: Error finding container 288c5427ebaab80fc1d1fe45bd10c44557dff114eed209f22cf64d45f927138d: Status 404 returned error can't find the container with id 288c5427ebaab80fc1d1fe45bd10c44557dff114eed209f22cf64d45f927138d
Oct 30 18:55:48.164995 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.164950    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:55:48.165084 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.165056    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:55:48.165133 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.165097    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:55:48.194079 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.194009    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:48.216233 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.216196    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" event={"ID":"4a3a967e-d63e-46e5-ba22-292d2c997aa6","Type":"ContainerStarted","Data":"288c5427ebaab80fc1d1fe45bd10c44557dff114eed209f22cf64d45f927138d"}
Oct 30 18:55:48.671145 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:55:48.671109    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200f5e70_1d1e_4f6e_9662_a7afd4e3dd7e.slice/crio-23c482d350730631e5e6d6f304252bc5330f56cb0b5c3073809bbd38736764d9 WatchSource:0}: Error finding container 23c482d350730631e5e6d6f304252bc5330f56cb0b5c3073809bbd38736764d9: Status 404 returned error can't find the container with id 23c482d350730631e5e6d6f304252bc5330f56cb0b5c3073809bbd38736764d9
Oct 30 18:55:48.672932 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.672887    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"]
Oct 30 18:55:48.678653 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.678606    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:55:48.678767 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.678705    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:55:48.678767 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:48.678751    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:55:49.250537 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:49.250473    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" event={"ID":"4a3a967e-d63e-46e5-ba22-292d2c997aa6","Type":"ContainerStarted","Data":"62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3"}
Oct 30 18:55:49.252601 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:49.252560    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" event={"ID":"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e","Type":"ContainerStarted","Data":"f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e"}
Oct 30 18:55:49.252714 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:49.252623    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" event={"ID":"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e","Type":"ContainerStarted","Data":"23c482d350730631e5e6d6f304252bc5330f56cb0b5c3073809bbd38736764d9"}
Oct 30 18:55:49.371296 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:49.371222    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" podStartSLOduration=2.371203641 podStartE2EDuration="2.371203641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:49.370900473 +0000 UTC m=+1040.258817565" watchObservedRunningTime="2026-10-30 18:55:49.371203641 +0000 UTC m=+1040.259120735"
Oct 30 18:55:49.499150 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:49.499089    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" podStartSLOduration=2.499072814 podStartE2EDuration="2.499072814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:49.497312418 +0000 UTC m=+1040.385229507" watchObservedRunningTime="2026-10-30 18:55:49.499072814 +0000 UTC m=+1040.386989903"
Oct 30 18:55:49.693323 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:49.693292    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:49.695125 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:49.695101    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:55:49.695249 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:49.695150    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 18:55:50.182313 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:55:50.182230    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:55:50.194376 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:50.194350    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:50.196011 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:50.195988    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:55:50.196125 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:50.196036    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 18:55:50.694102 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:50.694068    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:55:50.694518 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:50.694128    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 18:55:51.195528 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:51.194378    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:55:51.195528 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:51.194438    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 18:55:51.694061 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:51.694030    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:55:51.694229 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:51.694087    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 18:55:52.199581 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:52.198528    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:55:52.199581 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:52.198588    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 18:55:52.696540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:52.695875    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:55:52.696540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:52.695940    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 18:55:53.200425 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:53.198036    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:55:53.200425 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:53.198109    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 18:55:53.693717 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:53.693682    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:55:53.693912 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:53.693742    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 18:55:54.194283 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:54.194248    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:55:54.194778 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:54.194301    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 18:55:54.694077 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:54.694043    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 18:55:54.694287 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:54.694094    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 18:55:55.195566 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:55.195519    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 18:55:55.196103 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:55.195595    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 18:55:55.698721 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:55.698681    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:55.699107 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:55.699079    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:55.699964 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:55.699945    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 18:55:56.200163 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:56.200135    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:56.200647 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:56.200432    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:55:56.201089 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:55:56.201074    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 18:56:02.756515 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.744393    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"]
Oct 30 18:56:02.770764 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.770727    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:02.803406 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.803372    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"]
Oct 30 18:56:02.848404 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.848364    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-975cf5-openshift-default-dockercfg-g97ps\""
Oct 30 18:56:02.891525 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.884042    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:02.891849 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.891818    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7121f3-9966-49c8-9838-083262e11048-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:02.892043 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.892027    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7121f3-9966-49c8-9838-083262e11048-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:02.892230 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.892213    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:02.898682 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.898649    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:02.898980 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.898959    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428rs\" (UniqueName: \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-kube-api-access-428rs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:02.905209 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.903403    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:02.910533 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.905436    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:02.922738 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:02.922699    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.024767 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.024666    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.024767 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.024709    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.024767 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.024763    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428rs\" (UniqueName: \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-kube-api-access-428rs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.025076 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.024800    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.025076 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.024827    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.025076 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.024855    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.025076 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.024978    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.025076 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.025012    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7121f3-9966-49c8-9838-083262e11048-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.025076 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.025043    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7121f3-9966-49c8-9838-083262e11048-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.025409 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.025253    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.025671 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.025639    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.025893 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.025840    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.025893 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.025889    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.026241 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.026213    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7121f3-9966-49c8-9838-083262e11048-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.028009 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.027987    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7121f3-9966-49c8-9838-083262e11048-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.028160 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.028142    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.082446 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.082416    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-428rs\" (UniqueName: \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-kube-api-access-428rs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.083956 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.083923    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.094715 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.094684    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:03.575838 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.572040    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"]
Oct 30 18:56:03.579471 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.579424    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:56:03.579599 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.579561    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:56:03.579642 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:03.579609    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:56:04.402507 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:04.402457    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" event={"ID":"8d7121f3-9966-49c8-9838-083262e11048","Type":"ContainerStarted","Data":"35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946"}
Oct 30 18:56:04.402926 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:04.402527    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" event={"ID":"8d7121f3-9966-49c8-9838-083262e11048","Type":"ContainerStarted","Data":"cf7024dc26aa639485956b7c79cc6d6cff0d798e73214ecc6c39d74bae4bedce"}
Oct 30 18:56:04.570879 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:04.570826    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" podStartSLOduration=2.570805924 podStartE2EDuration="2.570805924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:04.56895807 +0000 UTC m=+1055.456875186" watchObservedRunningTime="2026-10-30 18:56:04.570805924 +0000 UTC m=+1055.458723015"
Oct 30 18:56:05.095656 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:05.095620    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:05.096852 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:05.096832    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:56:05.096933 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:05.096879    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 18:56:06.095941 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:06.095905    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:56:06.096372 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:06.095962    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 18:56:07.095781 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:07.095748    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:56:07.095968 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:07.095812    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 18:56:08.095456 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:08.095407    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:56:08.095916 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:08.095479    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 18:56:09.095196 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.095163    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:56:09.095369 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.095232    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 18:56:09.337993 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.337948    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"]
Oct 30 18:56:09.345701 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.345629    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:09.375764 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.375727    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1e703f6-pqhvd\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:09.376023 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.375986    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr86r\" (UniqueName: \"kubernetes.io/projected/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-kube-api-access-kr86r\") pod \"maas-api-revoke-sub-d1e703f6-pqhvd\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:09.376165 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.376136    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1e703f6-pqhvd\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:09.377341 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.377315    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"]
Oct 30 18:56:09.398034 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.398000    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:09.429796 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.429755    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:56:09.477611 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.477578    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1e703f6-pqhvd\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:09.477797 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.477726    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1e703f6-pqhvd\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:09.477797 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.477747    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr86r\" (UniqueName: \"kubernetes.io/projected/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-kube-api-access-kr86r\") pod \"maas-api-revoke-sub-d1e703f6-pqhvd\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:09.478238 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.478210    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1e703f6-pqhvd\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:09.480269 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.480244    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1e703f6-pqhvd\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:09.536324 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.536292    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr86r\" (UniqueName: \"kubernetes.io/projected/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-kube-api-access-kr86r\") pod \"maas-api-revoke-sub-d1e703f6-pqhvd\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:09.660117 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:09.660013    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:10.037786 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:10.037750    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"]
Oct 30 18:56:10.038619 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:56:10.038590    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bba0c3b_340e_45d5_ba1e_9b7c2666f309.slice/crio-891057ae71c6f9b58c0243cd43d5e0438ad16ffbf925f5def147ca73e1cc0713 WatchSource:0}: Error finding container 891057ae71c6f9b58c0243cd43d5e0438ad16ffbf925f5def147ca73e1cc0713: Status 404 returned error can't find the container with id 891057ae71c6f9b58c0243cd43d5e0438ad16ffbf925f5def147ca73e1cc0713
Oct 30 18:56:10.095504 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:10.095450    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:56:10.095680 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:10.095535    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 18:56:10.463669 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:10.463628    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd" event={"ID":"6bba0c3b-340e-45d5-ba1e-9b7c2666f309","Type":"ContainerStarted","Data":"4f442c342489bdeecc3d1b7e43d5ee65a7038a28d29bdbb4cc4089395b184142"}
Oct 30 18:56:10.464066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:10.463689    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd" event={"ID":"6bba0c3b-340e-45d5-ba1e-9b7c2666f309","Type":"ContainerStarted","Data":"891057ae71c6f9b58c0243cd43d5e0438ad16ffbf925f5def147ca73e1cc0713"}
Oct 30 18:56:10.576233 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:10.576112    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd" podStartSLOduration=1.57609412 podStartE2EDuration="1.57609412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:10.575919137 +0000 UTC m=+1061.463836226" watchObservedRunningTime="2026-10-30 18:56:10.57609412 +0000 UTC m=+1061.464011210"
Oct 30 18:56:11.101005 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:11.100969    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:11.101394 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:11.101370    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:11.102201 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:11.102185    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 18:56:11.471170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:11.471135    4267 generic.go:356] "Generic (PLEG): container finished" podID="6bba0c3b-340e-45d5-ba1e-9b7c2666f309" containerID="4f442c342489bdeecc3d1b7e43d5ee65a7038a28d29bdbb4cc4089395b184142" exitCode=0
Oct 30 18:56:11.471602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:11.471199    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd" event={"ID":"6bba0c3b-340e-45d5-ba1e-9b7c2666f309","Type":"ContainerDied","Data":"4f442c342489bdeecc3d1b7e43d5ee65a7038a28d29bdbb4cc4089395b184142"}
Oct 30 18:56:12.630332 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:12.630309    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:12.715486 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:12.715456    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-xks-ca\") pod \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") "
Oct 30 18:56:12.715667 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:12.715590    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-openshift-ca\") pod \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") "
Oct 30 18:56:12.715667 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:12.715625    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-kube-api-access-kr86r\" (UniqueName: \"kubernetes.io/projected/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-kube-api-access-kr86r\") pod \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\" (UID: \"6bba0c3b-340e-45d5-ba1e-9b7c2666f309\") "
Oct 30 18:56:12.715943 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:12.715911    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-openshift-ca" pod "6bba0c3b-340e-45d5-ba1e-9b7c2666f309" (UID: "6bba0c3b-340e-45d5-ba1e-9b7c2666f309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:12.717729 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:12.717703    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-xks-ca" pod "6bba0c3b-340e-45d5-ba1e-9b7c2666f309" (UID: "6bba0c3b-340e-45d5-ba1e-9b7c2666f309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:12.717897 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:12.717876    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-kube-api-access-kr86r" pod "6bba0c3b-340e-45d5-ba1e-9b7c2666f309" (UID: "6bba0c3b-340e-45d5-ba1e-9b7c2666f309"). InnerVolumeSpecName "kube-api-access-kr86r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:12.816604 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:12.816523    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:12.816604 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:12.816554    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr86r\" (UniqueName: \"kubernetes.io/projected/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-kube-api-access-kr86r\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:12.816604 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:12.816564    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bba0c3b-340e-45d5-ba1e-9b7c2666f309-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:13.484938 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:13.484860    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"
Oct 30 18:56:13.484938 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:13.484892    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd" event={"ID":"6bba0c3b-340e-45d5-ba1e-9b7c2666f309","Type":"ContainerDied","Data":"891057ae71c6f9b58c0243cd43d5e0438ad16ffbf925f5def147ca73e1cc0713"}
Oct 30 18:56:13.485112 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:13.484944    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="891057ae71c6f9b58c0243cd43d5e0438ad16ffbf925f5def147ca73e1cc0713"
Oct 30 18:56:17.117751 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.117715    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"]
Oct 30 18:56:17.118274 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.118227    4267 cpu_manager.go:414] "containerMap: removing container" podUID="6bba0c3b-340e-45d5-ba1e-9b7c2666f309" containerName="revoke-keys"
Oct 30 18:56:17.118274 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.118239    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bba0c3b-340e-45d5-ba1e-9b7c2666f309" containerName="revoke-keys"
Oct 30 18:56:17.118406 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.118330    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bba0c3b-340e-45d5-ba1e-9b7c2666f309" containerName="revoke-keys"
Oct 30 18:56:17.121579 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.121553    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.164917 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.164880    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.165117 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.164929    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.165183 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.165105    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gsb\" (UniqueName: \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-kube-api-access-x2gsb\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.165240 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.165190    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.165329 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.165312    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.165390 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.165345    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.165390 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.165383    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.165513 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.165456    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e4a69c-3d67-4022-af5b-e768f43f1d85-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.165579 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.165567    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.173672 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.173648    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f5763a-openshift-default-dockercfg-hrzwg\""
Oct 30 18:56:17.249895 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.249857    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"]
Oct 30 18:56:17.266739 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.266684    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.266934 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.266771    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gsb\" (UniqueName: \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-kube-api-access-x2gsb\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.266934 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.266831    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.267070 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.266963    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.267070 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.266998    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.267070 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.267037    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.267230 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.267076    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e4a69c-3d67-4022-af5b-e768f43f1d85-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.267230 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.267122    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.267230 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.267189    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.267230 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.267220    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.267778 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.267753    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.268171 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.267895    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e4a69c-3d67-4022-af5b-e768f43f1d85-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.268317 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.268113    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.268317 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.268170    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.269481 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.269454    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.270006 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.269978    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.335472 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.335435    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gsb\" (UniqueName: \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-kube-api-access-x2gsb\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.338776 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.338749    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.436628 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.436596    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:17.862821 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.862788    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"]
Oct 30 18:56:17.863732 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:56:17.863706    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e4a69c_3d67_4022_af5b_e768f43f1d85.slice/crio-3f70ea2aae1f106dae6faa581a50e79850f02f0e8e7bf52481c1a6da79205b0e WatchSource:0}: Error finding container 3f70ea2aae1f106dae6faa581a50e79850f02f0e8e7bf52481c1a6da79205b0e: Status 404 returned error can't find the container with id 3f70ea2aae1f106dae6faa581a50e79850f02f0e8e7bf52481c1a6da79205b0e
Oct 30 18:56:17.866567 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.866532    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:56:17.866681 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.866603    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:56:17.866681 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:17.866645    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:56:18.543629 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:18.543584    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" event={"ID":"92e4a69c-3d67-4022-af5b-e768f43f1d85","Type":"ContainerStarted","Data":"7faad4ec3fbd65b5688df1a928cd7f13b53fa1407ebd586a390269530005801a"}
Oct 30 18:56:18.544156 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:18.543656    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" event={"ID":"92e4a69c-3d67-4022-af5b-e768f43f1d85","Type":"ContainerStarted","Data":"3f70ea2aae1f106dae6faa581a50e79850f02f0e8e7bf52481c1a6da79205b0e"}
Oct 30 18:56:18.786787 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:18.786725    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" podStartSLOduration=1.7867103979999999 podStartE2EDuration="1.786710398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:18.735083515 +0000 UTC m=+1069.623000603" watchObservedRunningTime="2026-10-30 18:56:18.786710398 +0000 UTC m=+1069.674627487"
Oct 30 18:56:19.439312 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:19.439277    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:19.439639 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:19.439612    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:56:19.439836 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:19.439802    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:56:20.167005 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:56:20.166967    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:56:20.438516 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:20.437479    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:56:20.438516 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:20.437582    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:56:21.437960 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:21.437916    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:56:21.439286 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:21.439254    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:56:22.445517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:22.445024    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:56:22.445517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:22.445110    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:56:23.437790 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:23.437750    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:56:23.438046 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:23.438011    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:56:24.438317 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:24.437854    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:56:24.438317 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:24.437917    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:56:25.441680 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:25.441647    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:25.442163 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:25.442149    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:25.442945 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:25.442923    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 18:56:40.611258 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.611219    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"]
Oct 30 18:56:40.616450 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.616429    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:40.673143 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.673109    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"]
Oct 30 18:56:40.718143 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.718099    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a21455-rc7hs\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:40.718371 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.718234    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhjw\" (UniqueName: \"kubernetes.io/projected/defae603-d594-4eb2-b349-cc30ace02f2b-kube-api-access-khhjw\") pod \"maas-api-revoke-sub-01a21455-rc7hs\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:40.718535 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.718510    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a21455-rc7hs\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:40.718838 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.718813    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:40.718932 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.718884    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:56:40.819556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.819513    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a21455-rc7hs\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:40.819763 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.819597    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a21455-rc7hs\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:40.819763 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.819647    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhjw\" (UniqueName: \"kubernetes.io/projected/defae603-d594-4eb2-b349-cc30ace02f2b-kube-api-access-khhjw\") pod \"maas-api-revoke-sub-01a21455-rc7hs\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:40.820247 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.820219    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a21455-rc7hs\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:40.822086 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.822060    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a21455-rc7hs\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:40.876774 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.876706    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhjw\" (UniqueName: \"kubernetes.io/projected/defae603-d594-4eb2-b349-cc30ace02f2b-kube-api-access-khhjw\") pod \"maas-api-revoke-sub-01a21455-rc7hs\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:40.929825 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:40.929792    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:41.304065 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:41.304036    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"]
Oct 30 18:56:41.305169 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:56:41.305145    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefae603_d594_4eb2_b349_cc30ace02f2b.slice/crio-b815b2cd9f4949e0bf1cccbea49a90d896b7cf7d7c9d41d614a615ee0b7bcda6 WatchSource:0}: Error finding container b815b2cd9f4949e0bf1cccbea49a90d896b7cf7d7c9d41d614a615ee0b7bcda6: Status 404 returned error can't find the container with id b815b2cd9f4949e0bf1cccbea49a90d896b7cf7d7c9d41d614a615ee0b7bcda6
Oct 30 18:56:41.674485 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:41.674454    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"]
Oct 30 18:56:41.674929 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:41.674675    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy" containerID="cri-o://550aef35600f09143f4333aa4a7e77df3691abc58943c5fff28198426902e82d" gracePeriod=30
Oct 30 18:56:41.715682 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:41.715632    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs" event={"ID":"defae603-d594-4eb2-b349-cc30ace02f2b","Type":"ContainerStarted","Data":"6949803b214c0ef6164a235516e025d3c47e2795e7426a7ce56891a8505dc18d"}
Oct 30 18:56:41.715873 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:41.715706    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs" event={"ID":"defae603-d594-4eb2-b349-cc30ace02f2b","Type":"ContainerStarted","Data":"b815b2cd9f4949e0bf1cccbea49a90d896b7cf7d7c9d41d614a615ee0b7bcda6"}
Oct 30 18:56:41.726194 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:41.726161    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"]
Oct 30 18:56:41.726445 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:41.726419    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy" containerID="cri-o://94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb" gracePeriod=30
Oct 30 18:56:42.017971 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:42.017917    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs" podStartSLOduration=2.017903229 podStartE2EDuration="2.017903229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:42.01639117 +0000 UTC m=+1092.904308258" watchObservedRunningTime="2026-10-30 18:56:42.017903229 +0000 UTC m=+1092.905820366"
Oct 30 18:56:42.729675 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:42.729600    4267 generic.go:356] "Generic (PLEG): container finished" podID="defae603-d594-4eb2-b349-cc30ace02f2b" containerID="6949803b214c0ef6164a235516e025d3c47e2795e7426a7ce56891a8505dc18d" exitCode=0
Oct 30 18:56:42.730069 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:42.729693    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs" event={"ID":"defae603-d594-4eb2-b349-cc30ace02f2b","Type":"ContainerDied","Data":"6949803b214c0ef6164a235516e025d3c47e2795e7426a7ce56891a8505dc18d"}
Oct 30 18:56:44.780976 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:44.780881    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs" event={"ID":"defae603-d594-4eb2-b349-cc30ace02f2b","Type":"ContainerDied","Data":"b815b2cd9f4949e0bf1cccbea49a90d896b7cf7d7c9d41d614a615ee0b7bcda6"}
Oct 30 18:56:44.780976 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:44.780944    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b815b2cd9f4949e0bf1cccbea49a90d896b7cf7d7c9d41d614a615ee0b7bcda6"
Oct 30 18:56:44.799973 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:44.799935    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:44.902447 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:44.902415    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-openshift-ca\") pod \"defae603-d594-4eb2-b349-cc30ace02f2b\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") "
Oct 30 18:56:44.902644 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:44.902464    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-xks-ca\") pod \"defae603-d594-4eb2-b349-cc30ace02f2b\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") "
Oct 30 18:56:44.902723 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:44.902705    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defae603-d594-4eb2-b349-cc30ace02f2b-kube-api-access-khhjw\" (UniqueName: \"kubernetes.io/projected/defae603-d594-4eb2-b349-cc30ace02f2b-kube-api-access-khhjw\") pod \"defae603-d594-4eb2-b349-cc30ace02f2b\" (UID: \"defae603-d594-4eb2-b349-cc30ace02f2b\") "
Oct 30 18:56:44.902846 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:44.902827    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-openshift-ca" pod "defae603-d594-4eb2-b349-cc30ace02f2b" (UID: "defae603-d594-4eb2-b349-cc30ace02f2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:44.903119 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:44.903102    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:44.904543 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:44.904516    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-xks-ca" pod "defae603-d594-4eb2-b349-cc30ace02f2b" (UID: "defae603-d594-4eb2-b349-cc30ace02f2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:44.904670 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:44.904648    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defae603-d594-4eb2-b349-cc30ace02f2b-kube-api-access-khhjw" pod "defae603-d594-4eb2-b349-cc30ace02f2b" (UID: "defae603-d594-4eb2-b349-cc30ace02f2b"). InnerVolumeSpecName "kube-api-access-khhjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:45.004687 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:45.004602    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defae603-d594-4eb2-b349-cc30ace02f2b-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:45.004687 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:45.004637    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhjw\" (UniqueName: \"kubernetes.io/projected/defae603-d594-4eb2-b349-cc30ace02f2b-kube-api-access-khhjw\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:45.786308 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:45.786274    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"
Oct 30 18:56:46.793617 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:46.793580    4267 generic.go:356] "Generic (PLEG): container finished" podID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerID="550aef35600f09143f4333aa4a7e77df3691abc58943c5fff28198426902e82d" exitCode=0
Oct 30 18:56:46.793955 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:46.793647    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" event={"ID":"a54a5660-e61e-48fd-9d91-33e37acff5b4","Type":"ContainerDied","Data":"550aef35600f09143f4333aa4a7e77df3691abc58943c5fff28198426902e82d"}
Oct 30 18:56:47.013829 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.013800    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:56:47.030970 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.030933    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-certs\") pod \"a54a5660-e61e-48fd-9d91-33e37acff5b4\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") "
Oct 30 18:56:47.031138 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.030988    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-data\") pod \"a54a5660-e61e-48fd-9d91-33e37acff5b4\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") "
Oct 30 18:56:47.031138 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031035    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-socket\") pod \"a54a5660-e61e-48fd-9d91-33e37acff5b4\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") "
Oct 30 18:56:47.031138 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031079    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-token\" (UniqueName: \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-token\") pod \"a54a5660-e61e-48fd-9d91-33e37acff5b4\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") "
Oct 30 18:56:47.031316 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031148    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-credential-socket\") pod \"a54a5660-e61e-48fd-9d91-33e37acff5b4\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") "
Oct 30 18:56:47.031316 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031186    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54a5660-e61e-48fd-9d91-33e37acff5b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a54a5660-e61e-48fd-9d91-33e37acff5b4-istiod-ca-cert\") pod \"a54a5660-e61e-48fd-9d91-33e37acff5b4\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") "
Oct 30 18:56:47.031316 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031236    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-certs" pod "a54a5660-e61e-48fd-9d91-33e37acff5b4" (UID: "a54a5660-e61e-48fd-9d91-33e37acff5b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:47.031316 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031271    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-kube-api-access-t997c\" (UniqueName: \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-kube-api-access-t997c\") pod \"a54a5660-e61e-48fd-9d91-33e37acff5b4\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") "
Oct 30 18:56:47.031567 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031326    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-podinfo\") pod \"a54a5660-e61e-48fd-9d91-33e37acff5b4\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") "
Oct 30 18:56:47.031567 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031390    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-envoy\") pod \"a54a5660-e61e-48fd-9d91-33e37acff5b4\" (UID: \"a54a5660-e61e-48fd-9d91-33e37acff5b4\") "
Oct 30 18:56:47.031567 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031461    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-socket" pod "a54a5660-e61e-48fd-9d91-33e37acff5b4" (UID: "a54a5660-e61e-48fd-9d91-33e37acff5b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:47.031567 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031478    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-credential-socket" pod "a54a5660-e61e-48fd-9d91-33e37acff5b4" (UID: "a54a5660-e61e-48fd-9d91-33e37acff5b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:47.031833 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031653    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54a5660-e61e-48fd-9d91-33e37acff5b4-istiod-ca-cert" pod "a54a5660-e61e-48fd-9d91-33e37acff5b4" (UID: "a54a5660-e61e-48fd-9d91-33e37acff5b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:47.031948 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031883    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.031948 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031901    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a54a5660-e61e-48fd-9d91-33e37acff5b4-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.031948 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031917    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.031948 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.031931    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.032236 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.032203    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-data" pod "a54a5660-e61e-48fd-9d91-33e37acff5b4" (UID: "a54a5660-e61e-48fd-9d91-33e37acff5b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:47.034368 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.034338    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-envoy" pod "a54a5660-e61e-48fd-9d91-33e37acff5b4" (UID: "a54a5660-e61e-48fd-9d91-33e37acff5b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:47.034368 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.034357    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-token" pod "a54a5660-e61e-48fd-9d91-33e37acff5b4" (UID: "a54a5660-e61e-48fd-9d91-33e37acff5b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:47.036935 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.036910    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-kube-api-access-t997c" pod "a54a5660-e61e-48fd-9d91-33e37acff5b4" (UID: "a54a5660-e61e-48fd-9d91-33e37acff5b4"). InnerVolumeSpecName "kube-api-access-t997c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:47.037608 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.037582    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-podinfo" pod "a54a5660-e61e-48fd-9d91-33e37acff5b4" (UID: "a54a5660-e61e-48fd-9d91-33e37acff5b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:56:47.065285 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.065256    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:56:47.133146 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133112    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-credential-socket\") pod \"a63104ff-ed9d-4970-b1fe-06d50d28209f\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") "
Oct 30 18:56:47.133353 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133162    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-token\" (UniqueName: \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-token\") pod \"a63104ff-ed9d-4970-b1fe-06d50d28209f\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") "
Oct 30 18:56:47.133353 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133249    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-envoy\") pod \"a63104ff-ed9d-4970-b1fe-06d50d28209f\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") "
Oct 30 18:56:47.133353 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133283    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-data\") pod \"a63104ff-ed9d-4970-b1fe-06d50d28209f\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") "
Oct 30 18:56:47.133353 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133332    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63104ff-ed9d-4970-b1fe-06d50d28209f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63104ff-ed9d-4970-b1fe-06d50d28209f-istiod-ca-cert\") pod \"a63104ff-ed9d-4970-b1fe-06d50d28209f\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") "
Oct 30 18:56:47.133581 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133385    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-podinfo\") pod \"a63104ff-ed9d-4970-b1fe-06d50d28209f\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") "
Oct 30 18:56:47.133581 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133428    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-socket\") pod \"a63104ff-ed9d-4970-b1fe-06d50d28209f\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") "
Oct 30 18:56:47.133581 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133439    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-credential-socket" pod "a63104ff-ed9d-4970-b1fe-06d50d28209f" (UID: "a63104ff-ed9d-4970-b1fe-06d50d28209f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:47.133581 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133540    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-certs\") pod \"a63104ff-ed9d-4970-b1fe-06d50d28209f\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") "
Oct 30 18:56:47.133581 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133575    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-kube-api-access-79tf5\" (UniqueName: \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-kube-api-access-79tf5\") pod \"a63104ff-ed9d-4970-b1fe-06d50d28209f\" (UID: \"a63104ff-ed9d-4970-b1fe-06d50d28209f\") "
Oct 30 18:56:47.133824 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133667    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-data" pod "a63104ff-ed9d-4970-b1fe-06d50d28209f" (UID: "a63104ff-ed9d-4970-b1fe-06d50d28209f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:47.134001 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133961    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.134001 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133979    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.134001 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.133994    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.134186 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.134007    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.134186 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.134021    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t997c\" (UniqueName: \"kubernetes.io/projected/a54a5660-e61e-48fd-9d91-33e37acff5b4-kube-api-access-t997c\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.134186 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.134035    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.134186 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.134049    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a54a5660-e61e-48fd-9d91-33e37acff5b4-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.134386 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.134346    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-socket" pod "a63104ff-ed9d-4970-b1fe-06d50d28209f" (UID: "a63104ff-ed9d-4970-b1fe-06d50d28209f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:47.134598 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.134559    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-certs" pod "a63104ff-ed9d-4970-b1fe-06d50d28209f" (UID: "a63104ff-ed9d-4970-b1fe-06d50d28209f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:47.134742 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.134719    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63104ff-ed9d-4970-b1fe-06d50d28209f-istiod-ca-cert" pod "a63104ff-ed9d-4970-b1fe-06d50d28209f" (UID: "a63104ff-ed9d-4970-b1fe-06d50d28209f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:47.136449 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.136422    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-podinfo" pod "a63104ff-ed9d-4970-b1fe-06d50d28209f" (UID: "a63104ff-ed9d-4970-b1fe-06d50d28209f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:56:47.136561 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.136460    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-kube-api-access-79tf5" pod "a63104ff-ed9d-4970-b1fe-06d50d28209f" (UID: "a63104ff-ed9d-4970-b1fe-06d50d28209f"). InnerVolumeSpecName "kube-api-access-79tf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:47.136624 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.136599    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-token" pod "a63104ff-ed9d-4970-b1fe-06d50d28209f" (UID: "a63104ff-ed9d-4970-b1fe-06d50d28209f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:47.136830 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.136807    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-envoy" pod "a63104ff-ed9d-4970-b1fe-06d50d28209f" (UID: "a63104ff-ed9d-4970-b1fe-06d50d28209f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:47.239395 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.239116    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.239395 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.239160    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79tf5\" (UniqueName: \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-kube-api-access-79tf5\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.239395 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.239181    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.239395 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.239195    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.239395 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.239211    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63104ff-ed9d-4970-b1fe-06d50d28209f-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.239395 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.239225    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63104ff-ed9d-4970-b1fe-06d50d28209f-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.239395 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.239246    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63104ff-ed9d-4970-b1fe-06d50d28209f-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:47.801084 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.801044    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp" event={"ID":"a54a5660-e61e-48fd-9d91-33e37acff5b4","Type":"ContainerDied","Data":"d1465965ad41afb04c846e049a60de2a21edc7d4548119c5184ea2dd32f3cd4c"}
Oct 30 18:56:47.801533 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.801116    4267 scope.go:122] "RemoveContainer" containerID="550aef35600f09143f4333aa4a7e77df3691abc58943c5fff28198426902e82d"
Oct 30 18:56:47.801533 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.801051    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"
Oct 30 18:56:47.802433 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.802414    4267 generic.go:356] "Generic (PLEG): container finished" podID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerID="94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb" exitCode=0
Oct 30 18:56:47.802533 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.802481    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"
Oct 30 18:56:47.802533 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.802513    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" event={"ID":"a63104ff-ed9d-4970-b1fe-06d50d28209f","Type":"ContainerDied","Data":"94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb"}
Oct 30 18:56:47.802619 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.802570    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng" event={"ID":"a63104ff-ed9d-4970-b1fe-06d50d28209f","Type":"ContainerDied","Data":"174f8b8535fe6baff40aacbdeb9b32855d877ed6ee11d6d77bf687ad260b0f2a"}
Oct 30 18:56:47.813445 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.813426    4267 scope.go:122] "RemoveContainer" containerID="94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb"
Oct 30 18:56:47.822811 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.822788    4267 scope.go:122] "RemoveContainer" containerID="94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb"
Oct 30 18:56:47.823133 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:56:47.823113    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb\": container with ID starting with 94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb not found: ID does not exist" containerID="94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb"
Oct 30 18:56:47.823210 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.823146    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb"} err="failed to get container status \"94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb\": rpc error: code = NotFound desc = could not find container \"94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb\": container with ID starting with 94b358776c22791a1de20ca6cb49407eaeeb702258d6767fa8d69bd373a0a4cb not found: ID does not exist"
Oct 30 18:56:47.968919 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:47.968877    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"]
Oct 30 18:56:48.020703 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:48.020608    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-92mng"]
Oct 30 18:56:48.081465 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:48.081430    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"]
Oct 30 18:56:48.085604 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:48.085580    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-mctkp"]
Oct 30 18:56:49.220244 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:49.220209    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" path="/var/lib/kubelet/pods/a54a5660-e61e-48fd-9d91-33e37acff5b4/volumes"
Oct 30 18:56:49.220905 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:49.220883    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" path="/var/lib/kubelet/pods/a63104ff-ed9d-4970-b1fe-06d50d28209f/volumes"
Oct 30 18:56:50.166840 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:56:50.166797    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:56:51.314485 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.314453    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"]
Oct 30 18:56:51.314936 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.314920    4267 cpu_manager.go:414] "containerMap: removing container" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy"
Oct 30 18:56:51.314979 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.314936    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy"
Oct 30 18:56:51.314979 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.314951    4267 cpu_manager.go:414] "containerMap: removing container" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy"
Oct 30 18:56:51.314979 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.314957    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy"
Oct 30 18:56:51.314979 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.314978    4267 cpu_manager.go:414] "containerMap: removing container" podUID="defae603-d594-4eb2-b349-cc30ace02f2b" containerName="revoke-keys"
Oct 30 18:56:51.315102 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.314983    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defae603-d594-4eb2-b349-cc30ace02f2b" containerName="revoke-keys"
Oct 30 18:56:51.315102 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.315049    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54a5660-e61e-48fd-9d91-33e37acff5b4" containerName="istio-proxy"
Oct 30 18:56:51.315102 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.315057    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63104ff-ed9d-4970-b1fe-06d50d28209f" containerName="istio-proxy"
Oct 30 18:56:51.315102 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.315067    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="defae603-d594-4eb2-b349-cc30ace02f2b" containerName="revoke-keys"
Oct 30 18:56:51.319995 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.319973    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:51.374645 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.374610    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"]
Oct 30 18:56:51.393674 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.393635    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxsb2\" (UniqueName: \"kubernetes.io/projected/ee12d538-c02d-4c75-a2f7-1a654aaef531-kube-api-access-xxsb2\") pod \"maas-api-revoke-sub-dff8cc94-5wxnm\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:51.393849 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.393795    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff8cc94-5wxnm\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:51.393936 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.393876    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff8cc94-5wxnm\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:51.421048 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.421012    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:51.421237 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.421079    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:56:51.495140 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.495103    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff8cc94-5wxnm\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:51.495300 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.495163    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff8cc94-5wxnm\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:51.495300 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.495222    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxsb2\" (UniqueName: \"kubernetes.io/projected/ee12d538-c02d-4c75-a2f7-1a654aaef531-kube-api-access-xxsb2\") pod \"maas-api-revoke-sub-dff8cc94-5wxnm\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:51.495838 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.495817    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff8cc94-5wxnm\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:51.497743 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.497715    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff8cc94-5wxnm\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:51.552395 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.552360    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxsb2\" (UniqueName: \"kubernetes.io/projected/ee12d538-c02d-4c75-a2f7-1a654aaef531-kube-api-access-xxsb2\") pod \"maas-api-revoke-sub-dff8cc94-5wxnm\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:51.632517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:51.632417    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:52.012549 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:52.012512    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"]
Oct 30 18:56:52.013442 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:56:52.013416    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee12d538_c02d_4c75_a2f7_1a654aaef531.slice/crio-8076634bd11fc24977b8e41d9fea9c5ddfe51ca86ab09fce1025322f1e60a2dd WatchSource:0}: Error finding container 8076634bd11fc24977b8e41d9fea9c5ddfe51ca86ab09fce1025322f1e60a2dd: Status 404 returned error can't find the container with id 8076634bd11fc24977b8e41d9fea9c5ddfe51ca86ab09fce1025322f1e60a2dd
Oct 30 18:56:52.839506 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:52.839453    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm" event={"ID":"ee12d538-c02d-4c75-a2f7-1a654aaef531","Type":"ContainerStarted","Data":"8d3cd8c5dda41de1981b1568004866de65f3c4c056e6f4051339499c7cab27bc"}
Oct 30 18:56:52.839931 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:52.839523    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm" event={"ID":"ee12d538-c02d-4c75-a2f7-1a654aaef531","Type":"ContainerStarted","Data":"8076634bd11fc24977b8e41d9fea9c5ddfe51ca86ab09fce1025322f1e60a2dd"}
Oct 30 18:56:52.952381 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:52.952323    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm" podStartSLOduration=1.9523069020000001 podStartE2EDuration="1.952306902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:52.951059053 +0000 UTC m=+1103.838976144" watchObservedRunningTime="2026-10-30 18:56:52.952306902 +0000 UTC m=+1103.840223991"
Oct 30 18:56:53.847405 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:53.847368    4267 generic.go:356] "Generic (PLEG): container finished" podID="ee12d538-c02d-4c75-a2f7-1a654aaef531" containerID="8d3cd8c5dda41de1981b1568004866de65f3c4c056e6f4051339499c7cab27bc" exitCode=0
Oct 30 18:56:53.847829 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:53.847449    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm" event={"ID":"ee12d538-c02d-4c75-a2f7-1a654aaef531","Type":"ContainerDied","Data":"8d3cd8c5dda41de1981b1568004866de65f3c4c056e6f4051339499c7cab27bc"}
Oct 30 18:56:54.997061 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:54.997032    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:55.138782 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.138689    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-openshift-ca\") pod \"ee12d538-c02d-4c75-a2f7-1a654aaef531\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") "
Oct 30 18:56:55.138963 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.138826    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee12d538-c02d-4c75-a2f7-1a654aaef531-kube-api-access-xxsb2\" (UniqueName: \"kubernetes.io/projected/ee12d538-c02d-4c75-a2f7-1a654aaef531-kube-api-access-xxsb2\") pod \"ee12d538-c02d-4c75-a2f7-1a654aaef531\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") "
Oct 30 18:56:55.138963 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.138867    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-xks-ca\") pod \"ee12d538-c02d-4c75-a2f7-1a654aaef531\" (UID: \"ee12d538-c02d-4c75-a2f7-1a654aaef531\") "
Oct 30 18:56:55.139117 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.139087    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-openshift-ca" pod "ee12d538-c02d-4c75-a2f7-1a654aaef531" (UID: "ee12d538-c02d-4c75-a2f7-1a654aaef531"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:55.139275 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.139249    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:55.141537 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.141478    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee12d538-c02d-4c75-a2f7-1a654aaef531-kube-api-access-xxsb2" pod "ee12d538-c02d-4c75-a2f7-1a654aaef531" (UID: "ee12d538-c02d-4c75-a2f7-1a654aaef531"). InnerVolumeSpecName "kube-api-access-xxsb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:55.141633 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.141540    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-xks-ca" pod "ee12d538-c02d-4c75-a2f7-1a654aaef531" (UID: "ee12d538-c02d-4c75-a2f7-1a654aaef531"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:55.240565 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.240527    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxsb2\" (UniqueName: \"kubernetes.io/projected/ee12d538-c02d-4c75-a2f7-1a654aaef531-kube-api-access-xxsb2\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:55.240565 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.240559    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee12d538-c02d-4c75-a2f7-1a654aaef531-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:56:55.860194 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.860164    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"
Oct 30 18:56:55.860378 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.860211    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm" event={"ID":"ee12d538-c02d-4c75-a2f7-1a654aaef531","Type":"ContainerDied","Data":"8076634bd11fc24977b8e41d9fea9c5ddfe51ca86ab09fce1025322f1e60a2dd"}
Oct 30 18:56:55.860378 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:55.860261    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8076634bd11fc24977b8e41d9fea9c5ddfe51ca86ab09fce1025322f1e60a2dd"
Oct 30 18:56:58.250725 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.250689    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"]
Oct 30 18:56:58.251222 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.251153    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ee12d538-c02d-4c75-a2f7-1a654aaef531" containerName="revoke-keys"
Oct 30 18:56:58.251222 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.251165    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee12d538-c02d-4c75-a2f7-1a654aaef531" containerName="revoke-keys"
Oct 30 18:56:58.251331 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.251281    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee12d538-c02d-4c75-a2f7-1a654aaef531" containerName="revoke-keys"
Oct 30 18:56:58.254653 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.254633    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.358327 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.358287    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f5763a\"/\"distinct-w0-f5763a-kserve-self-signed-certs\""
Oct 30 18:56:58.361950 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.361931    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"]
Oct 30 18:56:58.378954 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.378927    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55ad1ee-81e9-444e-a078-7815d3295b4a-tls-certs\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.379072 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.378961    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-dshm\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.379109 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.379070    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-tmp-dir\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.379150 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.379113    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcc55\" (UniqueName: \"kubernetes.io/projected/c55ad1ee-81e9-444e-a078-7815d3295b4a-kube-api-access-fcc55\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.379252 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.379238    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-home\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.379284 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.379267    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-model-cache\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.480144 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.480106    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55ad1ee-81e9-444e-a078-7815d3295b4a-tls-certs\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.480349 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.480165    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-dshm\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.480349 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.480236    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-tmp-dir\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.480349 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.480279    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcc55\" (UniqueName: \"kubernetes.io/projected/c55ad1ee-81e9-444e-a078-7815d3295b4a-kube-api-access-fcc55\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.480551 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.480370    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-home\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.480551 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.480407    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-model-cache\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.480805 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.480752    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-tmp-dir\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.480949 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.480819    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-home\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.480949 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.480828    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-model-cache\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.482960 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.482940    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-dshm\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.483220 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.483197    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55ad1ee-81e9-444e-a078-7815d3295b4a-tls-certs\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.539933 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.539862    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcc55\" (UniqueName: \"kubernetes.io/projected/c55ad1ee-81e9-444e-a078-7815d3295b4a-kube-api-access-fcc55\") pod \"distinct-w0-f5763a-kserve-cc589d985-5h8b8\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") " pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.567624 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.567586    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:56:58.964654 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:58.964297    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"]
Oct 30 18:56:59.884475 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:59.884433    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" event={"ID":"c55ad1ee-81e9-444e-a078-7815d3295b4a","Type":"ContainerStarted","Data":"62fb6e2f4e3f5df49fe573bc984b126785eb62f9b395464e61f9085377aad8eb"}
Oct 30 18:56:59.884475 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:59.884480    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" event={"ID":"c55ad1ee-81e9-444e-a078-7815d3295b4a","Type":"ContainerStarted","Data":"735442d218bf44d2e76dc050c35ddeb09220de651100472cd8cda5d720773c17"}
Oct 30 18:56:59.998609 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:56:59.998554    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podStartSLOduration=1.998541358 podStartE2EDuration="1.998541358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:59.997447102 +0000 UTC m=+1110.885364214" watchObservedRunningTime="2026-10-30 18:56:59.998541358 +0000 UTC m=+1110.886458446"
Oct 30 18:57:00.568694 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:00.568655    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:57:02.963314 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:02.963275    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"]
Oct 30 18:57:02.967152 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:02.967132    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.059181 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.059140    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"]
Oct 30 18:57:03.135617 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.135578    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.135836 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.135633    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.135836 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.135767    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.135836 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.135824    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.135979 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.135891    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5117062a-c82a-41d6-8286-526d69c1504c-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.136021 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.136003    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.136060 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.136044    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5117062a-c82a-41d6-8286-526d69c1504c-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.136221 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.136195    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.136291 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.136242    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h2v5\" (UniqueName: \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-kube-api-access-8h2v5\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.237649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.237558    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.237649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.237602    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h2v5\" (UniqueName: \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-kube-api-access-8h2v5\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.237649 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.237643    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.237942 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.237679    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.237942 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.237715    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.237942 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.237744    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.237942 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.237763    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5117062a-c82a-41d6-8286-526d69c1504c-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.237942 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.237830    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.237942 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.237869    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5117062a-c82a-41d6-8286-526d69c1504c-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.238239 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.238044    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-credential-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.238239 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.238169    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-certs\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.238239 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.238201    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-data\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.238410 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.238260    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-socket\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.238545 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.238525    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5117062a-c82a-41d6-8286-526d69c1504c-istiod-ca-cert\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.240065 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.240037    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-envoy\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.240783 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.240761    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5117062a-c82a-41d6-8286-526d69c1504c-istio-podinfo\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.294653 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.294622    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-istio-token\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.294907 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.294880    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h2v5\" (UniqueName: \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-kube-api-access-8h2v5\") pod \"e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") " pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:03.580669 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:03.580589    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:04.012665 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:04.012629    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"]
Oct 30 18:57:04.051271 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:57:04.051238    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5117062a_c82a_41d6_8286_526d69c1504c.slice/crio-99bcbe9f63b78b1bf48afd0cb6a8779875c9c8d48f621cfc9798208a70341a66 WatchSource:0}: Error finding container 99bcbe9f63b78b1bf48afd0cb6a8779875c9c8d48f621cfc9798208a70341a66: Status 404 returned error can't find the container with id 99bcbe9f63b78b1bf48afd0cb6a8779875c9c8d48f621cfc9798208a70341a66
Oct 30 18:57:04.053786 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:04.053744    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:57:04.053904 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:04.053823    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:57:04.053904 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:04.053851    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:57:04.939559 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:04.939513    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" event={"ID":"5117062a-c82a-41d6-8286-526d69c1504c","Type":"ContainerStarted","Data":"41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b"}
Oct 30 18:57:04.939774 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:04.939575    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" event={"ID":"5117062a-c82a-41d6-8286-526d69c1504c","Type":"ContainerStarted","Data":"99bcbe9f63b78b1bf48afd0cb6a8779875c9c8d48f621cfc9798208a70341a66"}
Oct 30 18:57:05.063842 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:05.063792    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" podStartSLOduration=3.06377844 podStartE2EDuration="3.06377844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:05.062141467 +0000 UTC m=+1115.950058590" watchObservedRunningTime="2026-10-30 18:57:05.06377844 +0000 UTC m=+1115.951695530"
Oct 30 18:57:05.581087 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:05.581052    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:05.582601 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:05.582575    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:05.582727 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:05.582625    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" podUID="5117062a-c82a-41d6-8286-526d69c1504c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:57:06.581861 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:06.581820    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:06.582287 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:06.581877    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" podUID="5117062a-c82a-41d6-8286-526d69c1504c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:57:07.581566 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:07.581483    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:07.581748 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:07.581586    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" podUID="5117062a-c82a-41d6-8286-526d69c1504c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:57:08.568661 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:08.568623    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:57:08.581640 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:08.581613    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:57:08.581809 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:08.581608    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:08.581809 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:08.581693    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" podUID="5117062a-c82a-41d6-8286-526d69c1504c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:57:08.974991 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:08.974962    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 18:57:09.581036 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:09.581001    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:09.581414 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:09.581060    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" podUID="5117062a-c82a-41d6-8286-526d69c1504c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:57:10.581975 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:10.581938    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:57:10.582368 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:10.581996    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" podUID="5117062a-c82a-41d6-8286-526d69c1504c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 18:57:11.586938 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:11.586841    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:11.587429 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:11.587234    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:11.587960 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:11.587942    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 18:57:20.158853 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:57:20.158812    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:57:37.167371 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.167335    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"]
Oct 30 18:57:37.172407 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.172386    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:37.226550 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.226517    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:37.275834 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.275802    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:57:37.279142 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.279118    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"]
Oct 30 18:57:37.358170 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.358133    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8kgw\" (UniqueName: \"kubernetes.io/projected/2a1cf202-2a0f-4012-9d41-95f65077333b-kube-api-access-v8kgw\") pod \"maas-api-revoke-sub-61d277d1-88svd\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:37.358362 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.358306    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d277d1-88svd\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:37.358477 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.358462    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d277d1-88svd\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:37.459370 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.459331    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d277d1-88svd\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:37.459607 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.459452    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d277d1-88svd\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:37.459607 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.459489    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kgw\" (UniqueName: \"kubernetes.io/projected/2a1cf202-2a0f-4012-9d41-95f65077333b-kube-api-access-v8kgw\") pod \"maas-api-revoke-sub-61d277d1-88svd\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:37.460341 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.460312    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d277d1-88svd\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:37.461927 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.461900    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d277d1-88svd\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:37.517595 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.517556    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kgw\" (UniqueName: \"kubernetes.io/projected/2a1cf202-2a0f-4012-9d41-95f65077333b-kube-api-access-v8kgw\") pod \"maas-api-revoke-sub-61d277d1-88svd\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:37.786191 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:37.786107    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:38.164953 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:38.164926    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"]
Oct 30 18:57:38.166731 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:57:38.166687    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1cf202_2a0f_4012_9d41_95f65077333b.slice/crio-3f4f14e1118c65e7d044369883ac6955b691f16b9ce2f9b098fa3cc97ca3c5b9 WatchSource:0}: Error finding container 3f4f14e1118c65e7d044369883ac6955b691f16b9ce2f9b098fa3cc97ca3c5b9: Status 404 returned error can't find the container with id 3f4f14e1118c65e7d044369883ac6955b691f16b9ce2f9b098fa3cc97ca3c5b9
Oct 30 18:57:38.196647 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:38.196612    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd" event={"ID":"2a1cf202-2a0f-4012-9d41-95f65077333b","Type":"ContainerStarted","Data":"3f4f14e1118c65e7d044369883ac6955b691f16b9ce2f9b098fa3cc97ca3c5b9"}
Oct 30 18:57:39.205462 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:39.205425    4267 generic.go:356] "Generic (PLEG): container finished" podID="2a1cf202-2a0f-4012-9d41-95f65077333b" containerID="41858990bea9c7afde62e92298d002af3397cb117683b88fed01788a566cd3b4" exitCode=0
Oct 30 18:57:39.205913 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:39.205526    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd" event={"ID":"2a1cf202-2a0f-4012-9d41-95f65077333b","Type":"ContainerDied","Data":"41858990bea9c7afde62e92298d002af3397cb117683b88fed01788a566cd3b4"}
Oct 30 18:57:40.367887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.367862    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:40.413301 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.413269    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1cf202-2a0f-4012-9d41-95f65077333b-kube-api-access-v8kgw\" (UniqueName: \"kubernetes.io/projected/2a1cf202-2a0f-4012-9d41-95f65077333b-kube-api-access-v8kgw\") pod \"2a1cf202-2a0f-4012-9d41-95f65077333b\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") "
Oct 30 18:57:40.413541 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.413325    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-openshift-ca\") pod \"2a1cf202-2a0f-4012-9d41-95f65077333b\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") "
Oct 30 18:57:40.413541 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.413457    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-xks-ca\") pod \"2a1cf202-2a0f-4012-9d41-95f65077333b\" (UID: \"2a1cf202-2a0f-4012-9d41-95f65077333b\") "
Oct 30 18:57:40.413781 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.413748    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-openshift-ca" pod "2a1cf202-2a0f-4012-9d41-95f65077333b" (UID: "2a1cf202-2a0f-4012-9d41-95f65077333b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:40.413957 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.413934    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:40.416737 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.416707    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1cf202-2a0f-4012-9d41-95f65077333b-kube-api-access-v8kgw" pod "2a1cf202-2a0f-4012-9d41-95f65077333b" (UID: "2a1cf202-2a0f-4012-9d41-95f65077333b"). InnerVolumeSpecName "kube-api-access-v8kgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:40.416900 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.416874    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-xks-ca" pod "2a1cf202-2a0f-4012-9d41-95f65077333b" (UID: "2a1cf202-2a0f-4012-9d41-95f65077333b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:40.515176 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.515077    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1cf202-2a0f-4012-9d41-95f65077333b-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:40.515176 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.515118    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8kgw\" (UniqueName: \"kubernetes.io/projected/2a1cf202-2a0f-4012-9d41-95f65077333b-kube-api-access-v8kgw\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:40.698023 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.697986    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"]
Oct 30 18:57:40.698550 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.698529    4267 cpu_manager.go:414] "containerMap: removing container" podUID="2a1cf202-2a0f-4012-9d41-95f65077333b" containerName="revoke-keys"
Oct 30 18:57:40.698550 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.698547    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1cf202-2a0f-4012-9d41-95f65077333b" containerName="revoke-keys"
Oct 30 18:57:40.698753 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.698662    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1cf202-2a0f-4012-9d41-95f65077333b" containerName="revoke-keys"
Oct 30 18:57:40.702027 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.702004    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:40.717302 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.717270    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d7d05d-g8lct\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:40.717431 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.717322    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d7d05d-g8lct\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:40.717540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.717478    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cttqz\" (UniqueName: \"kubernetes.io/projected/38c757c1-1c84-46d2-9b95-40f0942d766d-kube-api-access-cttqz\") pod \"maas-api-revoke-sub-59d7d05d-g8lct\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:40.760928 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.760892    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"]
Oct 30 18:57:40.819219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.819127    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d7d05d-g8lct\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:40.819219 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.819176    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d7d05d-g8lct\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:40.819468 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.819240    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttqz\" (UniqueName: \"kubernetes.io/projected/38c757c1-1c84-46d2-9b95-40f0942d766d-kube-api-access-cttqz\") pod \"maas-api-revoke-sub-59d7d05d-g8lct\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:40.819904 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.819880    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d7d05d-g8lct\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:40.821657 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.821632    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d7d05d-g8lct\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:40.876810 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:40.876780    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cttqz\" (UniqueName: \"kubernetes.io/projected/38c757c1-1c84-46d2-9b95-40f0942d766d-kube-api-access-cttqz\") pod \"maas-api-revoke-sub-59d7d05d-g8lct\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:41.018011 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.017974    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:41.222328 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.222288    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"
Oct 30 18:57:41.222328 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.222314    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd" event={"ID":"2a1cf202-2a0f-4012-9d41-95f65077333b","Type":"ContainerDied","Data":"3f4f14e1118c65e7d044369883ac6955b691f16b9ce2f9b098fa3cc97ca3c5b9"}
Oct 30 18:57:41.222605 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.222375    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f4f14e1118c65e7d044369883ac6955b691f16b9ce2f9b098fa3cc97ca3c5b9"
Oct 30 18:57:41.337264 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.337182    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"]
Oct 30 18:57:41.342257 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.342240    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.393057 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.393029    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"]
Oct 30 18:57:41.394199 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:57:41.394131    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c757c1_1c84_46d2_9b95_40f0942d766d.slice/crio-39aa5e6e42ff04df8065d243965d3701c681563d0ad4c3592e9475346e411ee5 WatchSource:0}: Error finding container 39aa5e6e42ff04df8065d243965d3701c681563d0ad4c3592e9475346e411ee5: Status 404 returned error can't find the container with id 39aa5e6e42ff04df8065d243965d3701c681563d0ad4c3592e9475346e411ee5
Oct 30 18:57:41.394372 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.394354    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-611053\"/\"distinct-w5-611053-kserve-self-signed-certs\""
Oct 30 18:57:41.399115 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.399034    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"]
Oct 30 18:57:41.425602 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.425564    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-home\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.425710 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.425612    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5c5k\" (UniqueName: \"kubernetes.io/projected/78bb5520-dfd6-49c9-8506-0922401452bd-kube-api-access-n5c5k\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.425775 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.425755    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78bb5520-dfd6-49c9-8506-0922401452bd-tls-certs\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.425926 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.425908    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-tmp-dir\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.425997 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.425948    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-model-cache\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.426056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.426041    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-dshm\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.527254 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.527222    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-home\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.527437 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.527260    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5c5k\" (UniqueName: \"kubernetes.io/projected/78bb5520-dfd6-49c9-8506-0922401452bd-kube-api-access-n5c5k\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.527437 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.527323    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78bb5520-dfd6-49c9-8506-0922401452bd-tls-certs\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.527437 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.527407    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-tmp-dir\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.527622 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.527441    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-model-cache\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.527622 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.527504    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-dshm\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.527819 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.527794    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-home\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.527884 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.527830    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-tmp-dir\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.527884 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.527845    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-model-cache\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.529797 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.529777    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-dshm\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.530062 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.530040    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78bb5520-dfd6-49c9-8506-0922401452bd-tls-certs\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.584430 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.584389    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5c5k\" (UniqueName: \"kubernetes.io/projected/78bb5520-dfd6-49c9-8506-0922401452bd-kube-api-access-n5c5k\") pod \"distinct-w5-611053-kserve-cff86dc44-flms7\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") " pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:41.664184 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:41.664149    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:42.066143 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:42.066113    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"]
Oct 30 18:57:42.066517 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:57:42.066478    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78bb5520_dfd6_49c9_8506_0922401452bd.slice/crio-6c5d5f46a148bf0df4f653f2ce31a9601230f615f4de0f7c387a20efe5dd7690 WatchSource:0}: Error finding container 6c5d5f46a148bf0df4f653f2ce31a9601230f615f4de0f7c387a20efe5dd7690: Status 404 returned error can't find the container with id 6c5d5f46a148bf0df4f653f2ce31a9601230f615f4de0f7c387a20efe5dd7690
Oct 30 18:57:42.230360 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:42.230324    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct" event={"ID":"38c757c1-1c84-46d2-9b95-40f0942d766d","Type":"ContainerStarted","Data":"398ccdf2c5d0da34bd32cf2dd715d4b22c69b63a8248d986106dda1a5b120c25"}
Oct 30 18:57:42.230566 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:42.230385    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct" event={"ID":"38c757c1-1c84-46d2-9b95-40f0942d766d","Type":"ContainerStarted","Data":"39aa5e6e42ff04df8065d243965d3701c681563d0ad4c3592e9475346e411ee5"}
Oct 30 18:57:42.232016 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:42.231986    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" event={"ID":"78bb5520-dfd6-49c9-8506-0922401452bd","Type":"ContainerStarted","Data":"4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f"}
Oct 30 18:57:42.232121 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:42.232024    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" event={"ID":"78bb5520-dfd6-49c9-8506-0922401452bd","Type":"ContainerStarted","Data":"6c5d5f46a148bf0df4f653f2ce31a9601230f615f4de0f7c387a20efe5dd7690"}
Oct 30 18:57:42.345686 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:42.345585    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct" podStartSLOduration=2.345571614 podStartE2EDuration="2.345571614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:42.34368382 +0000 UTC m=+1153.231600912" watchObservedRunningTime="2026-10-30 18:57:42.345571614 +0000 UTC m=+1153.233488745"
Oct 30 18:57:42.460480 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:42.460426    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podStartSLOduration=1.460408026 podStartE2EDuration="1.460408026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:42.457808125 +0000 UTC m=+1153.345725214" watchObservedRunningTime="2026-10-30 18:57:42.460408026 +0000 UTC m=+1153.348325118"
Oct 30 18:57:42.664729 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:42.664630    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:43.241068 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:43.241033    4267 generic.go:356] "Generic (PLEG): container finished" podID="38c757c1-1c84-46d2-9b95-40f0942d766d" containerID="398ccdf2c5d0da34bd32cf2dd715d4b22c69b63a8248d986106dda1a5b120c25" exitCode=0
Oct 30 18:57:43.241237 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:43.241164    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct" event={"ID":"38c757c1-1c84-46d2-9b95-40f0942d766d","Type":"ContainerDied","Data":"398ccdf2c5d0da34bd32cf2dd715d4b22c69b63a8248d986106dda1a5b120c25"}
Oct 30 18:57:44.387166 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.387141    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:44.459192 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.459160    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-xks-ca\") pod \"38c757c1-1c84-46d2-9b95-40f0942d766d\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") "
Oct 30 18:57:44.459376 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.459215    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-openshift-ca\") pod \"38c757c1-1c84-46d2-9b95-40f0942d766d\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") "
Oct 30 18:57:44.459376 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.459306    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c757c1-1c84-46d2-9b95-40f0942d766d-kube-api-access-cttqz\" (UniqueName: \"kubernetes.io/projected/38c757c1-1c84-46d2-9b95-40f0942d766d-kube-api-access-cttqz\") pod \"38c757c1-1c84-46d2-9b95-40f0942d766d\" (UID: \"38c757c1-1c84-46d2-9b95-40f0942d766d\") "
Oct 30 18:57:44.459702 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.459666    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-openshift-ca" pod "38c757c1-1c84-46d2-9b95-40f0942d766d" (UID: "38c757c1-1c84-46d2-9b95-40f0942d766d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:44.459894 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.459875    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:44.461396 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.461373    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-xks-ca" pod "38c757c1-1c84-46d2-9b95-40f0942d766d" (UID: "38c757c1-1c84-46d2-9b95-40f0942d766d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:44.461396 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.461387    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c757c1-1c84-46d2-9b95-40f0942d766d-kube-api-access-cttqz" pod "38c757c1-1c84-46d2-9b95-40f0942d766d" (UID: "38c757c1-1c84-46d2-9b95-40f0942d766d"). InnerVolumeSpecName "kube-api-access-cttqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:44.492723 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.492688    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"]
Oct 30 18:57:44.493436 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.493412    4267 cpu_manager.go:414] "containerMap: removing container" podUID="38c757c1-1c84-46d2-9b95-40f0942d766d" containerName="revoke-keys"
Oct 30 18:57:44.493436 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.493437    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c757c1-1c84-46d2-9b95-40f0942d766d" containerName="revoke-keys"
Oct 30 18:57:44.493614 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.493586    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c757c1-1c84-46d2-9b95-40f0942d766d" containerName="revoke-keys"
Oct 30 18:57:44.497399 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.497371    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.544232 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.544196    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"]
Oct 30 18:57:44.555891 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.555857    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bc98e1\"/\"kube-root-ca.crt\""
Oct 30 18:57:44.555891 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.555883    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bc98e1\"/\"default-dockercfg-hc99l\""
Oct 30 18:57:44.556135 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.555874    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bc98e1\"/\"openshift-service-ca.crt\""
Oct 30 18:57:44.556135 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.555986    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bc98e1\"/\"distinct-w3-bc98e1-kserve-self-signed-certs\""
Oct 30 18:57:44.561512 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.561470    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-model-cache\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.561676 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.561643    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c695\" (UniqueName: \"kubernetes.io/projected/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-kube-api-access-7c695\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.561747 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.561685    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tmp-dir\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.561747 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.561704    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-dshm\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.561859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.561799    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tls-certs\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.561859 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.561832    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-home\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.561967 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.561896    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c757c1-1c84-46d2-9b95-40f0942d766d-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:44.561967 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.561908    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cttqz\" (UniqueName: \"kubernetes.io/projected/38c757c1-1c84-46d2-9b95-40f0942d766d-kube-api-access-cttqz\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:44.663394 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.663360    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-home\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.663609 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.663446    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-model-cache\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.663710 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.663647    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c695\" (UniqueName: \"kubernetes.io/projected/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-kube-api-access-7c695\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.663769 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.663708    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tmp-dir\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.663769 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.663740    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-dshm\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.663867 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.663782    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tls-certs\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.663867 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.663826    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-home\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.663867 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.663856    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-model-cache\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.664035 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.663992    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tmp-dir\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.665993 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.665972    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-dshm\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.666282 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.666263    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tls-certs\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.721693 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.721663    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c695\" (UniqueName: \"kubernetes.io/projected/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-kube-api-access-7c695\") pod \"distinct-w3-bc98e1-kserve-d489758db-4h8sq\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") " pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:44.815193 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:44.815102    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:45.204056 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:45.204031    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"]
Oct 30 18:57:45.204449 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:57:45.204422    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e86f312_519f_4842_8e30_1c3a1aaf4ae7.slice/crio-3e5d63b99084fdb6d75e4303dfb605f48e631f731b56d969b3d60742af97ff78 WatchSource:0}: Error finding container 3e5d63b99084fdb6d75e4303dfb605f48e631f731b56d969b3d60742af97ff78: Status 404 returned error can't find the container with id 3e5d63b99084fdb6d75e4303dfb605f48e631f731b56d969b3d60742af97ff78
Oct 30 18:57:45.257081 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:45.257052    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" event={"ID":"6e86f312-519f-4842-8e30-1c3a1aaf4ae7","Type":"ContainerStarted","Data":"3e5d63b99084fdb6d75e4303dfb605f48e631f731b56d969b3d60742af97ff78"}
Oct 30 18:57:45.258528 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:45.258478    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct" event={"ID":"38c757c1-1c84-46d2-9b95-40f0942d766d","Type":"ContainerDied","Data":"39aa5e6e42ff04df8065d243965d3701c681563d0ad4c3592e9475346e411ee5"}
Oct 30 18:57:45.258621 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:45.258539    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"
Oct 30 18:57:45.258674 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:45.258545    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39aa5e6e42ff04df8065d243965d3701c681563d0ad4c3592e9475346e411ee5"
Oct 30 18:57:46.266216 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:46.266180    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" event={"ID":"6e86f312-519f-4842-8e30-1c3a1aaf4ae7","Type":"ContainerStarted","Data":"ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8"}
Oct 30 18:57:46.384247 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:46.384193    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podStartSLOduration=2.3841788839999998 podStartE2EDuration="2.384178884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:46.38126028 +0000 UTC m=+1157.269177371" watchObservedRunningTime="2026-10-30 18:57:46.384178884 +0000 UTC m=+1157.272095964"
Oct 30 18:57:46.815852 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:46.815825    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:50.166627 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:57:50.166583    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:57:51.664403 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.664368    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:51.677207 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.677168    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:51.782618 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.782588    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"]
Oct 30 18:57:51.786079 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.786063    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:51.832693 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.832654    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"]
Oct 30 18:57:51.839391 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.839359    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:51.843420 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.841350    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:57:51.866236 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.866199    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32f757f5-sw665\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:51.866401 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.866358    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smd6l\" (UniqueName: \"kubernetes.io/projected/90f739ba-8f4a-4076-a7eb-17d5d12323b6-kube-api-access-smd6l\") pod \"maas-api-revoke-sub-32f757f5-sw665\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:51.866514 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.866479    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32f757f5-sw665\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:51.967424 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.967387    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smd6l\" (UniqueName: \"kubernetes.io/projected/90f739ba-8f4a-4076-a7eb-17d5d12323b6-kube-api-access-smd6l\") pod \"maas-api-revoke-sub-32f757f5-sw665\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:51.967654 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.967476    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32f757f5-sw665\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:51.967654 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.967593    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32f757f5-sw665\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:51.968389 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.968359    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32f757f5-sw665\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:51.969861 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:51.969835    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32f757f5-sw665\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:52.025833 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:52.025797    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smd6l\" (UniqueName: \"kubernetes.io/projected/90f739ba-8f4a-4076-a7eb-17d5d12323b6-kube-api-access-smd6l\") pod \"maas-api-revoke-sub-32f757f5-sw665\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:52.099967 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:52.099936    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:52.316747 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:52.316657    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 18:57:52.524683 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:52.524648    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"]
Oct 30 18:57:52.527344 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:57:52.527315    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f739ba_8f4a_4076_a7eb_17d5d12323b6.slice/crio-c929a83f9e9b4d236d26d6f9c6c302e35082e40e69c0f0b61bb7d61997fe70a0 WatchSource:0}: Error finding container c929a83f9e9b4d236d26d6f9c6c302e35082e40e69c0f0b61bb7d61997fe70a0: Status 404 returned error can't find the container with id c929a83f9e9b4d236d26d6f9c6c302e35082e40e69c0f0b61bb7d61997fe70a0
Oct 30 18:57:53.312816 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:53.312775    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665" event={"ID":"90f739ba-8f4a-4076-a7eb-17d5d12323b6","Type":"ContainerStarted","Data":"fcc3bfa7f210cefbc9a78cfef6d7c632ce6da0a4562b6c759e02c43ad60804e7"}
Oct 30 18:57:53.313279 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:53.312846    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665" event={"ID":"90f739ba-8f4a-4076-a7eb-17d5d12323b6","Type":"ContainerStarted","Data":"c929a83f9e9b4d236d26d6f9c6c302e35082e40e69c0f0b61bb7d61997fe70a0"}
Oct 30 18:57:53.429575 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:53.429479    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665" podStartSLOduration=2.429459724 podStartE2EDuration="2.429459724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:53.426410614 +0000 UTC m=+1164.314327704" watchObservedRunningTime="2026-10-30 18:57:53.429459724 +0000 UTC m=+1164.317376815"
Oct 30 18:57:54.321937 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:54.321901    4267 generic.go:356] "Generic (PLEG): container finished" podID="90f739ba-8f4a-4076-a7eb-17d5d12323b6" containerID="fcc3bfa7f210cefbc9a78cfef6d7c632ce6da0a4562b6c759e02c43ad60804e7" exitCode=0
Oct 30 18:57:54.322387 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:54.321979    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665" event={"ID":"90f739ba-8f4a-4076-a7eb-17d5d12323b6","Type":"ContainerDied","Data":"fcc3bfa7f210cefbc9a78cfef6d7c632ce6da0a4562b6c759e02c43ad60804e7"}
Oct 30 18:57:54.815556 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:54.815523    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:54.828829 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:54.828800    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:55.347951 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.347919    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 18:57:55.539642 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.539604    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:55.627572 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.626983    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-xks-ca\") pod \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") "
Oct 30 18:57:55.627572 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.627132    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f739ba-8f4a-4076-a7eb-17d5d12323b6-kube-api-access-smd6l\" (UniqueName: \"kubernetes.io/projected/90f739ba-8f4a-4076-a7eb-17d5d12323b6-kube-api-access-smd6l\") pod \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") "
Oct 30 18:57:55.627572 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.627183    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-openshift-ca\") pod \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\" (UID: \"90f739ba-8f4a-4076-a7eb-17d5d12323b6\") "
Oct 30 18:57:55.631514 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.628210    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-openshift-ca" pod "90f739ba-8f4a-4076-a7eb-17d5d12323b6" (UID: "90f739ba-8f4a-4076-a7eb-17d5d12323b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:55.635514 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.632049    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-xks-ca" pod "90f739ba-8f4a-4076-a7eb-17d5d12323b6" (UID: "90f739ba-8f4a-4076-a7eb-17d5d12323b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:55.635514 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.634091    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f739ba-8f4a-4076-a7eb-17d5d12323b6-kube-api-access-smd6l" pod "90f739ba-8f4a-4076-a7eb-17d5d12323b6" (UID: "90f739ba-8f4a-4076-a7eb-17d5d12323b6"). InnerVolumeSpecName "kube-api-access-smd6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:55.740517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.730414    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smd6l\" (UniqueName: \"kubernetes.io/projected/90f739ba-8f4a-4076-a7eb-17d5d12323b6-kube-api-access-smd6l\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:55.740517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.730451    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:55.740517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:55.730466    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f739ba-8f4a-4076-a7eb-17d5d12323b6-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:57:56.344831 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:56.344797    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"
Oct 30 18:57:56.344831 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:56.344803    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665" event={"ID":"90f739ba-8f4a-4076-a7eb-17d5d12323b6","Type":"ContainerDied","Data":"c929a83f9e9b4d236d26d6f9c6c302e35082e40e69c0f0b61bb7d61997fe70a0"}
Oct 30 18:57:56.345100 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:57:56.344868    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c929a83f9e9b4d236d26d6f9c6c302e35082e40e69c0f0b61bb7d61997fe70a0"
Oct 30 18:58:20.159684 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:58:20.159644    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:58:32.845522 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.845450    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"]
Oct 30 18:58:32.845980 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.845964    4267 cpu_manager.go:414] "containerMap: removing container" podUID="90f739ba-8f4a-4076-a7eb-17d5d12323b6" containerName="revoke-keys"
Oct 30 18:58:32.845980 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.845979    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f739ba-8f4a-4076-a7eb-17d5d12323b6" containerName="revoke-keys"
Oct 30 18:58:32.846079 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.846067    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f739ba-8f4a-4076-a7eb-17d5d12323b6" containerName="revoke-keys"
Oct 30 18:58:32.849527 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.849480    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:32.928241 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.928187    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:32.928438 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.928319    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:32.928438 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.928396    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:32.928438 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.928416    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:32.928666 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.928572    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:32.928730 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.928667    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:32.928730 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.928702    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkfzs\" (UniqueName: \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-kube-api-access-bkfzs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:32.928839 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.928809    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:32.928839 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.928833    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:32.957918 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:32.957878    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"]
Oct 30 18:58:33.029558 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029479    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.029558 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029552    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkfzs\" (UniqueName: \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-kube-api-access-bkfzs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.029822 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029622    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.029822 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029644    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.029822 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029691    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.029822 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029755    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.030040 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029854    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.030040 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029888    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.030040 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029952    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.030040 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029964    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.030040 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.029982    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.030293 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.030238    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.030376 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.030353    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.030653 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.030625    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.032362 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.032333    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.032504 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.032472    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.086289 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.086250    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.086441 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.086380    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkfzs\" (UniqueName: \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-kube-api-access-bkfzs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.164225 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.164134    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:33.596370 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.596339    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"]
Oct 30 18:58:33.596544 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:58:33.596523    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbdd267_f2ff_41fa_9394_4342eef03d4f.slice/crio-f294ef536edffc5fb803db47f8578d6f39cdbd2af2cf9880089bc3b5088fa4e0 WatchSource:0}: Error finding container f294ef536edffc5fb803db47f8578d6f39cdbd2af2cf9880089bc3b5088fa4e0: Status 404 returned error can't find the container with id f294ef536edffc5fb803db47f8578d6f39cdbd2af2cf9880089bc3b5088fa4e0
Oct 30 18:58:33.598902 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.598871    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:58:33.598972 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.598938    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:58:33.598972 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.598967    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:58:33.620199 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:33.620169    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" event={"ID":"bcbdd267-f2ff-41fa-9394-4342eef03d4f","Type":"ContainerStarted","Data":"f294ef536edffc5fb803db47f8578d6f39cdbd2af2cf9880089bc3b5088fa4e0"}
Oct 30 18:58:34.648388 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:34.648338    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" event={"ID":"bcbdd267-f2ff-41fa-9394-4342eef03d4f","Type":"ContainerStarted","Data":"e3315d47eeffd515b5e3973693d7669ac05908bce54cbf6f13519ef507e22753"}
Oct 30 18:58:35.164448 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:35.164405    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:35.165688 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:35.165658    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:58:35.165835 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:35.165713    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:58:36.165243 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:36.165205    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:58:36.165676 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:36.165270    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:58:37.165191 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:37.165103    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:58:37.165402 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:37.165231    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:58:38.164807 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:38.164765    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:58:38.164996 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:38.164826    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:58:39.164746 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:39.164714    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:58:39.165214 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:39.165189    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:58:40.165103 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:40.165068    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 18:58:40.165599 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:40.165129    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 18:58:41.170392 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:41.170303    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:41.170828 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:41.170715    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:41.171603 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:41.171580    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 18:58:41.339896 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:41.339839    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" podStartSLOduration=9.33982351 podStartE2EDuration="9.33982351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:34.82656486 +0000 UTC m=+1205.714481950" watchObservedRunningTime="2026-10-30 18:58:41.33982351 +0000 UTC m=+1212.227740658"
Oct 30 18:58:50.158800 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:58:50.158758    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:58:55.563550 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.563459    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"]
Oct 30 18:58:55.567606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.567584    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:55.624472 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.624441    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"]
Oct 30 18:58:55.639811 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.639777    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed98e73a-zdbj9\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:55.639988 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.639856    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed98e73a-zdbj9\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:55.640030 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.639973    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrkg\" (UniqueName: \"kubernetes.io/projected/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-kube-api-access-zsrkg\") pod \"maas-api-revoke-sub-ed98e73a-zdbj9\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:55.669038 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.669006    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:55.669224 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.669102    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:58:55.741109 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.741073    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed98e73a-zdbj9\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:55.741317 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.741132    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed98e73a-zdbj9\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:55.741317 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.741161    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrkg\" (UniqueName: \"kubernetes.io/projected/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-kube-api-access-zsrkg\") pod \"maas-api-revoke-sub-ed98e73a-zdbj9\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:55.741837 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.741813    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed98e73a-zdbj9\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:55.743529 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.743510    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed98e73a-zdbj9\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:55.798150 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.798112    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrkg\" (UniqueName: \"kubernetes.io/projected/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-kube-api-access-zsrkg\") pod \"maas-api-revoke-sub-ed98e73a-zdbj9\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:55.881825 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:55.881742    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:56.270050 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:58:56.270017    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6d0fe0_0ad4_4d5a_9f48_68c545841baf.slice/crio-f5c6689dc028b03a6d7ce36abd39a729eea59d1848df50c69320a24c7b6530a4 WatchSource:0}: Error finding container f5c6689dc028b03a6d7ce36abd39a729eea59d1848df50c69320a24c7b6530a4: Status 404 returned error can't find the container with id f5c6689dc028b03a6d7ce36abd39a729eea59d1848df50c69320a24c7b6530a4
Oct 30 18:58:56.271589 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:56.271566    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"]
Oct 30 18:58:56.800693 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:56.800649    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9" event={"ID":"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf","Type":"ContainerStarted","Data":"fcef0a3c712c1674bce2c34e702390e15b65b870d6c3436d50bbaf3e10d73d1b"}
Oct 30 18:58:56.801110 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:56.800748    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9" event={"ID":"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf","Type":"ContainerStarted","Data":"f5c6689dc028b03a6d7ce36abd39a729eea59d1848df50c69320a24c7b6530a4"}
Oct 30 18:58:56.921003 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:56.920897    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9" podStartSLOduration=1.920881657 podStartE2EDuration="1.920881657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:56.918114094 +0000 UTC m=+1227.806031184" watchObservedRunningTime="2026-10-30 18:58:56.920881657 +0000 UTC m=+1227.808798743"
Oct 30 18:58:57.808606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:57.808570    4267 generic.go:356] "Generic (PLEG): container finished" podID="bd6d0fe0-0ad4-4d5a-9f48-68c545841baf" containerID="fcef0a3c712c1674bce2c34e702390e15b65b870d6c3436d50bbaf3e10d73d1b" exitCode=0
Oct 30 18:58:57.809024 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:57.808661    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9" event={"ID":"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf","Type":"ContainerDied","Data":"fcef0a3c712c1674bce2c34e702390e15b65b870d6c3436d50bbaf3e10d73d1b"}
Oct 30 18:58:58.957392 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:58.957363    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:59.102196 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.102088    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-openshift-ca\") pod \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") "
Oct 30 18:58:59.102386 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.102262    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-kube-api-access-zsrkg\" (UniqueName: \"kubernetes.io/projected/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-kube-api-access-zsrkg\") pod \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") "
Oct 30 18:58:59.102755 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.102394    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-xks-ca\") pod \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\" (UID: \"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf\") "
Oct 30 18:58:59.102755 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.102477    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-openshift-ca" pod "bd6d0fe0-0ad4-4d5a-9f48-68c545841baf" (UID: "bd6d0fe0-0ad4-4d5a-9f48-68c545841baf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:59.102897 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.102862    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:59.104407 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.104378    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-xks-ca" pod "bd6d0fe0-0ad4-4d5a-9f48-68c545841baf" (UID: "bd6d0fe0-0ad4-4d5a-9f48-68c545841baf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:59.104873 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.104854    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-kube-api-access-zsrkg" pod "bd6d0fe0-0ad4-4d5a-9f48-68c545841baf" (UID: "bd6d0fe0-0ad4-4d5a-9f48-68c545841baf"). InnerVolumeSpecName "kube-api-access-zsrkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:59.204274 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.204243    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsrkg\" (UniqueName: \"kubernetes.io/projected/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-kube-api-access-zsrkg\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:59.204274 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.204272    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:58:59.824430 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.824394    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"
Oct 30 18:58:59.824639 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.824421    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9" event={"ID":"bd6d0fe0-0ad4-4d5a-9f48-68c545841baf","Type":"ContainerDied","Data":"f5c6689dc028b03a6d7ce36abd39a729eea59d1848df50c69320a24c7b6530a4"}
Oct 30 18:58:59.824639 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:58:59.824482    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5c6689dc028b03a6d7ce36abd39a729eea59d1848df50c69320a24c7b6530a4"
Oct 30 18:59:09.256374 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.256336    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"]
Oct 30 18:59:09.256872 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.256838    4267 cpu_manager.go:414] "containerMap: removing container" podUID="bd6d0fe0-0ad4-4d5a-9f48-68c545841baf" containerName="revoke-keys"
Oct 30 18:59:09.256872 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.256852    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6d0fe0-0ad4-4d5a-9f48-68c545841baf" containerName="revoke-keys"
Oct 30 18:59:09.256944 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.256939    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6d0fe0-0ad4-4d5a-9f48-68c545841baf" containerName="revoke-keys"
Oct 30 18:59:09.260385 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.260365    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.314270 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.314240    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1290a6\"/\"kube-root-ca.crt\""
Oct 30 18:59:09.314466 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.314362    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1290a6\"/\"openshift-service-ca.crt\""
Oct 30 18:59:09.363866 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.363842    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1290a6\"/\"default-dockercfg-5658b\""
Oct 30 18:59:09.363866 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.363842    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1290a6\"/\"distinct-w4-1290a6-kserve-self-signed-certs\""
Oct 30 18:59:09.368613 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.368587    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"]
Oct 30 18:59:09.429161 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.429120    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tls-certs\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.429367 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.429330    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-home\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.429428 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.429389    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm6s7\" (UniqueName: \"kubernetes.io/projected/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-kube-api-access-rm6s7\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.429528 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.429434    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-dshm\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.429528 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.429469    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-model-cache\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.429644 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.429525    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tmp-dir\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.531150 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.531056    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm6s7\" (UniqueName: \"kubernetes.io/projected/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-kube-api-access-rm6s7\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.531150 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.531105    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-dshm\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.531150 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.531137    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-model-cache\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.531485 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.531164    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tmp-dir\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.531485 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.531209    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tls-certs\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.531485 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.531377    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-home\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.531767 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.531742    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-model-cache\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.531837 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.531769    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tmp-dir\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.531837 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.531812    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-home\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.533699 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.533675    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-dshm\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.534058 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.534038    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tls-certs\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.588243 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.588210    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm6s7\" (UniqueName: \"kubernetes.io/projected/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-kube-api-access-rm6s7\") pod \"distinct-w4-1290a6-kserve-75d56d9d7b-jpwml\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") " pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:09.872317 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:09.872230    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:10.248867 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:59:10.248837    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode44de0b9_15e4_4b2c_91ba_3f1d57f198bc.slice/crio-9c5fa0f062510c46e77d2a009605864370d1a94ab604b21c848bec12ebb377f8 WatchSource:0}: Error finding container 9c5fa0f062510c46e77d2a009605864370d1a94ab604b21c848bec12ebb377f8: Status 404 returned error can't find the container with id 9c5fa0f062510c46e77d2a009605864370d1a94ab604b21c848bec12ebb377f8
Oct 30 18:59:10.249620 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.249596    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"]
Oct 30 18:59:10.417680 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.417642    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"]
Oct 30 18:59:10.421869 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.421850    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.446128 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.446104    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsfv4\" (UniqueName: \"kubernetes.io/projected/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-kube-api-access-hsfv4\") pod \"maas-api-revoke-sub-80ac7525-8gz9z\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.446282 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.446143    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ac7525-8gz9z\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.446357 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.446284    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ac7525-8gz9z\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.478601 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.478571    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"]
Oct 30 18:59:10.522704 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.522625    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:10.522704 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.522677    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:59:10.547576 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.547544    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsfv4\" (UniqueName: \"kubernetes.io/projected/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-kube-api-access-hsfv4\") pod \"maas-api-revoke-sub-80ac7525-8gz9z\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.547718 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.547586    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ac7525-8gz9z\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.547718 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.547664    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ac7525-8gz9z\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.548365 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.548341    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ac7525-8gz9z\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.550128 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.550106    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ac7525-8gz9z\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.604374 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.604346    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsfv4\" (UniqueName: \"kubernetes.io/projected/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-kube-api-access-hsfv4\") pod \"maas-api-revoke-sub-80ac7525-8gz9z\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.735887 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.735851    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:10.902969 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.902931    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" event={"ID":"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc","Type":"ContainerStarted","Data":"38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136"}
Oct 30 18:59:10.902969 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:10.902976    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" event={"ID":"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc","Type":"ContainerStarted","Data":"9c5fa0f062510c46e77d2a009605864370d1a94ab604b21c848bec12ebb377f8"}
Oct 30 18:59:11.019903 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:11.019853    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podStartSLOduration=2.019838595 podStartE2EDuration="2.019838595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:11.016175992 +0000 UTC m=+1241.904093081" watchObservedRunningTime="2026-10-30 18:59:11.019838595 +0000 UTC m=+1241.907755684"
Oct 30 18:59:11.116823 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:59:11.116792    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe7c242_5a25_4a5e_be24_ce5c6b7d08e8.slice/crio-729cfc3dd487f1d33ba1b5d7971b306416cd986e21ec5d92882b37261edc4997 WatchSource:0}: Error finding container 729cfc3dd487f1d33ba1b5d7971b306416cd986e21ec5d92882b37261edc4997: Status 404 returned error can't find the container with id 729cfc3dd487f1d33ba1b5d7971b306416cd986e21ec5d92882b37261edc4997
Oct 30 18:59:11.118132 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:11.118100    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"]
Oct 30 18:59:11.873023 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:11.872986    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:11.911853 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:11.911803    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z" event={"ID":"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8","Type":"ContainerStarted","Data":"24d192343064371ec5bc2ac69a39b7d83f239c5ca898442fe3ad3ce708d5ae0f"}
Oct 30 18:59:11.912049 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:11.911864    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z" event={"ID":"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8","Type":"ContainerStarted","Data":"729cfc3dd487f1d33ba1b5d7971b306416cd986e21ec5d92882b37261edc4997"}
Oct 30 18:59:12.028641 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:12.028576    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z" podStartSLOduration=2.028560676 podStartE2EDuration="2.028560676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:12.02513771 +0000 UTC m=+1242.913054997" watchObservedRunningTime="2026-10-30 18:59:12.028560676 +0000 UTC m=+1242.916477764"
Oct 30 18:59:12.924053 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:12.924018    4267 generic.go:356] "Generic (PLEG): container finished" podID="1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8" containerID="24d192343064371ec5bc2ac69a39b7d83f239c5ca898442fe3ad3ce708d5ae0f" exitCode=0
Oct 30 18:59:12.924463 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:12.924108    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z" event={"ID":"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8","Type":"ContainerDied","Data":"24d192343064371ec5bc2ac69a39b7d83f239c5ca898442fe3ad3ce708d5ae0f"}
Oct 30 18:59:14.090232 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.090205    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:14.181711 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.181617    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-openshift-ca\") pod \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") "
Oct 30 18:59:14.181711 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.181671    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-xks-ca\") pod \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") "
Oct 30 18:59:14.182002 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.181979    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-openshift-ca" pod "1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8" (UID: "1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:14.183900 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.183862    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-xks-ca" pod "1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8" (UID: "1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:14.282558 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.282515    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-kube-api-access-hsfv4\" (UniqueName: \"kubernetes.io/projected/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-kube-api-access-hsfv4\") pod \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\" (UID: \"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8\") "
Oct 30 18:59:14.283055 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.283037    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:59:14.283156 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.283060    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:59:14.284788 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.284758    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-kube-api-access-hsfv4" pod "1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8" (UID: "1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8"). InnerVolumeSpecName "kube-api-access-hsfv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:14.383743 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.383703    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsfv4\" (UniqueName: \"kubernetes.io/projected/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8-kube-api-access-hsfv4\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:59:14.938221 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.938187    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"
Oct 30 18:59:14.938221 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.938210    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z" event={"ID":"1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8","Type":"ContainerDied","Data":"729cfc3dd487f1d33ba1b5d7971b306416cd986e21ec5d92882b37261edc4997"}
Oct 30 18:59:14.938452 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:14.938268    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="729cfc3dd487f1d33ba1b5d7971b306416cd986e21ec5d92882b37261edc4997"
Oct 30 18:59:19.872411 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:19.872375    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:19.889514 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:19.888617    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:19.998576 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:19.998461    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 18:59:20.178860 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:59:20.178761    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:59:33.006263 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.006218    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"]
Oct 30 18:59:33.006932 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.006910    4267 cpu_manager.go:414] "containerMap: removing container" podUID="1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8" containerName="revoke-keys"
Oct 30 18:59:33.006932 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.006932    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8" containerName="revoke-keys"
Oct 30 18:59:33.007105 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.007092    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8" containerName="revoke-keys"
Oct 30 18:59:33.011766 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.011750    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.087138 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.087099    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.087305 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.087150    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.087305 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.087207    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de3ab02-b017-4de5-87c0-d102e55164fd-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.087305 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.087283    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.087463 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.087361    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql84n\" (UniqueName: \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-kube-api-access-ql84n\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.087463 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.087399    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.087463 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.087436    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.087606 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.087548    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.087690 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.087668    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.116042 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.115998    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"]
Oct 30 18:59:33.189124 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189084    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189306 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189143    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189347 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189310    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de3ab02-b017-4de5-87c0-d102e55164fd-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189430 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189409    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189476 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189444    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql84n\" (UniqueName: \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-kube-api-access-ql84n\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189547 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189487    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189547 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189538    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189655 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189563    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189655 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189647    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189802 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189781    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189888 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189795    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.189888 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.189845    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.190094 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.190070    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de3ab02-b017-4de5-87c0-d102e55164fd-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.190216 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.190198    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.191889 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.191860    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.192129 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.192111    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.246232 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.246198    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.247052 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.247025    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql84n\" (UniqueName: \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-kube-api-access-ql84n\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.324620 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.324532    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:33.802718 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.802687    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"]
Oct 30 18:59:33.802940 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:59:33.802918    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de3ab02_b017_4de5_87c0_d102e55164fd.slice/crio-3c7abe5d1dbdf6e74f9e6c1d80188979009755f28f06ccff98a1ee668e6295dd WatchSource:0}: Error finding container 3c7abe5d1dbdf6e74f9e6c1d80188979009755f28f06ccff98a1ee668e6295dd: Status 404 returned error can't find the container with id 3c7abe5d1dbdf6e74f9e6c1d80188979009755f28f06ccff98a1ee668e6295dd
Oct 30 18:59:33.805487 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.805453    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:59:33.805566 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.805554    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:59:33.805599 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:33.805586    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:59:34.157851 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:34.157752    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" event={"ID":"3de3ab02-b017-4de5-87c0-d102e55164fd","Type":"ContainerStarted","Data":"27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30"}
Oct 30 18:59:34.157851 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:34.157818    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" event={"ID":"3de3ab02-b017-4de5-87c0-d102e55164fd","Type":"ContainerStarted","Data":"3c7abe5d1dbdf6e74f9e6c1d80188979009755f28f06ccff98a1ee668e6295dd"}
Oct 30 18:59:34.274473 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:34.274410    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" podStartSLOduration=2.274391424 podStartE2EDuration="2.274391424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:34.274144421 +0000 UTC m=+1265.162061510" watchObservedRunningTime="2026-10-30 18:59:34.274391424 +0000 UTC m=+1265.162308518"
Oct 30 18:59:34.327199 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:34.327166    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:34.333517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:34.329875    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:59:34.333517 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:34.329939    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:59:35.330518 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:35.324937    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:59:35.330518 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:35.325004    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:59:36.330518 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:36.329961    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:59:36.330518 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:36.330032    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:59:37.333521 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:37.327599    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:59:37.333521 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:37.327661    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:59:38.333111 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:38.333070    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:59:38.333416 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:38.333382    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:59:39.325525 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:39.325015    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:59:39.325525 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:39.325076    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:59:40.325196 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:40.325156    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 18:59:40.325559 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:40.325221    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 18:59:41.330683 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:41.330644    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:41.331185 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:41.331161    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:41.332044 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:41.332023    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 18:59:50.158659 ip-10-0-131-68 kubenswrapper[4267]: E1030 18:59:50.158614    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 18:59:54.646345 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.646294    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"]
Oct 30 18:59:54.651095 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.651068    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:54.705442 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.705402    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"]
Oct 30 18:59:54.744782 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.744732    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1702ab-n7vnq\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:54.744994 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.744970    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6whtz\" (UniqueName: \"kubernetes.io/projected/5b102a36-afea-450a-b0b7-d757364ebeda-kube-api-access-6whtz\") pod \"maas-api-revoke-sub-4b1702ab-n7vnq\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:54.745134 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.745106    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1702ab-n7vnq\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:54.750854 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.750827    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:54.750854 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.750850    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 18:59:54.847022 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.846959    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6whtz\" (UniqueName: \"kubernetes.io/projected/5b102a36-afea-450a-b0b7-d757364ebeda-kube-api-access-6whtz\") pod \"maas-api-revoke-sub-4b1702ab-n7vnq\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:54.847205 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.847055    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1702ab-n7vnq\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:54.847205 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.847158    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1702ab-n7vnq\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:54.847807 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.847778    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1702ab-n7vnq\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:54.849506 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.849467    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1702ab-n7vnq\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:54.907540 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.907449    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6whtz\" (UniqueName: \"kubernetes.io/projected/5b102a36-afea-450a-b0b7-d757364ebeda-kube-api-access-6whtz\") pod \"maas-api-revoke-sub-4b1702ab-n7vnq\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:54.971066 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:54.971033    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:55.348216 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:55.348185    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"]
Oct 30 18:59:55.348644 ip-10-0-131-68 kubenswrapper[4267]: W1030 18:59:55.348618    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b102a36_afea_450a_b0b7_d757364ebeda.slice/crio-116263a89e5ceb1b567c9a05278c35c8ee65eac9911a797e9cd4e0f292d0fa76 WatchSource:0}: Error finding container 116263a89e5ceb1b567c9a05278c35c8ee65eac9911a797e9cd4e0f292d0fa76: Status 404 returned error can't find the container with id 116263a89e5ceb1b567c9a05278c35c8ee65eac9911a797e9cd4e0f292d0fa76
Oct 30 18:59:56.345516 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:56.345450    4267 generic.go:356] "Generic (PLEG): container finished" podID="5b102a36-afea-450a-b0b7-d757364ebeda" containerID="05b1918547bc14c227c6e8123d3ceb3900660f26db4eed01d99338933593a0fd" exitCode=0
Oct 30 18:59:56.345900 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:56.345586    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq" event={"ID":"5b102a36-afea-450a-b0b7-d757364ebeda","Type":"ContainerDied","Data":"05b1918547bc14c227c6e8123d3ceb3900660f26db4eed01d99338933593a0fd"}
Oct 30 18:59:56.345900 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:56.345641    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq" event={"ID":"5b102a36-afea-450a-b0b7-d757364ebeda","Type":"ContainerStarted","Data":"116263a89e5ceb1b567c9a05278c35c8ee65eac9911a797e9cd4e0f292d0fa76"}
Oct 30 18:59:57.499563 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:57.499538    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:57.683800 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:57.683773    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b102a36-afea-450a-b0b7-d757364ebeda-kube-api-access-6whtz\" (UniqueName: \"kubernetes.io/projected/5b102a36-afea-450a-b0b7-d757364ebeda-kube-api-access-6whtz\") pod \"5b102a36-afea-450a-b0b7-d757364ebeda\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") "
Oct 30 18:59:57.683945 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:57.683825    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-xks-ca\") pod \"5b102a36-afea-450a-b0b7-d757364ebeda\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") "
Oct 30 18:59:57.683995 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:57.683978    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-openshift-ca\") pod \"5b102a36-afea-450a-b0b7-d757364ebeda\" (UID: \"5b102a36-afea-450a-b0b7-d757364ebeda\") "
Oct 30 18:59:57.684315 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:57.684292    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-openshift-ca" pod "5b102a36-afea-450a-b0b7-d757364ebeda" (UID: "5b102a36-afea-450a-b0b7-d757364ebeda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:57.684562 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:57.684541    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:59:57.685933 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:57.685911    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-xks-ca" pod "5b102a36-afea-450a-b0b7-d757364ebeda" (UID: "5b102a36-afea-450a-b0b7-d757364ebeda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:57.686046 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:57.686030    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b102a36-afea-450a-b0b7-d757364ebeda-kube-api-access-6whtz" pod "5b102a36-afea-450a-b0b7-d757364ebeda" (UID: "5b102a36-afea-450a-b0b7-d757364ebeda"). InnerVolumeSpecName "kube-api-access-6whtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:57.785688 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:57.785652    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6whtz\" (UniqueName: \"kubernetes.io/projected/5b102a36-afea-450a-b0b7-d757364ebeda-kube-api-access-6whtz\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:59:57.785688 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:57.785682    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b102a36-afea-450a-b0b7-d757364ebeda-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 18:59:58.360435 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:58.360404    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"
Oct 30 18:59:58.360638 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:58.360403    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq" event={"ID":"5b102a36-afea-450a-b0b7-d757364ebeda","Type":"ContainerDied","Data":"116263a89e5ceb1b567c9a05278c35c8ee65eac9911a797e9cd4e0f292d0fa76"}
Oct 30 18:59:58.360638 ip-10-0-131-68 kubenswrapper[4267]: I1030 18:59:58.360550    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="116263a89e5ceb1b567c9a05278c35c8ee65eac9911a797e9cd4e0f292d0fa76"
Oct 30 19:00:00.163554 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.163471    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"]
Oct 30 19:00:00.164314 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.164292    4267 cpu_manager.go:414] "containerMap: removing container" podUID="5b102a36-afea-450a-b0b7-d757364ebeda" containerName="revoke-keys"
Oct 30 19:00:00.164382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.164316    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b102a36-afea-450a-b0b7-d757364ebeda" containerName="revoke-keys"
Oct 30 19:00:00.164462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.164449    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b102a36-afea-450a-b0b7-d757364ebeda" containerName="revoke-keys"
Oct 30 19:00:00.168323 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.168300    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"]
Oct 30 19:00:00.168464 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.168444    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"
Oct 30 19:00:00.172272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.172239    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"]
Oct 30 19:00:00.172400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.172346    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"
Oct 30 19:00:00.176677 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.176657    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"
Oct 30 19:00:00.211929 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.211888    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsp7r\" (UniqueName: \"kubernetes.io/projected/19b5b0a1-ea85-49d5-96a1-b417620ffacb-kube-api-access-wsp7r\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5\" (UID: \"19b5b0a1-ea85-49d5-96a1-b417620ffacb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"
Oct 30 19:00:00.212102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.212003    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b5b0a1-ea85-49d5-96a1-b417620ffacb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5\" (UID: \"19b5b0a1-ea85-49d5-96a1-b417620ffacb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"
Oct 30 19:00:00.212162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.212101    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw\" (UID: \"dd2f7f58-73ee-4675-9d18-b6d8595e2af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"
Oct 30 19:00:00.212162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.212129    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls\" (UID: \"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"
Oct 30 19:00:00.212270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.212210    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlkbn\" (UniqueName: \"kubernetes.io/projected/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-kube-api-access-qlkbn\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw\" (UID: \"dd2f7f58-73ee-4675-9d18-b6d8595e2af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"
Oct 30 19:00:00.212270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.212242    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9svr9\" (UniqueName: \"kubernetes.io/projected/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-kube-api-access-9svr9\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls\" (UID: \"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"
Oct 30 19:00:00.212839 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.212817    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"]
Oct 30 19:00:00.216930 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.216912    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"
Oct 30 19:00:00.220592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.220572    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:00:00.267212 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.267180    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"]
Oct 30 19:00:00.271451 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.271423    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"]
Oct 30 19:00:00.276783 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.276755    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"]
Oct 30 19:00:00.314075 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.314041    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw\" (UID: \"dd2f7f58-73ee-4675-9d18-b6d8595e2af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"
Oct 30 19:00:00.315426 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.315360    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls\" (UID: \"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"
Oct 30 19:00:00.316328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.316308    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlkbn\" (UniqueName: \"kubernetes.io/projected/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-kube-api-access-qlkbn\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw\" (UID: \"dd2f7f58-73ee-4675-9d18-b6d8595e2af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"
Oct 30 19:00:00.316802 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.316777    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9svr9\" (UniqueName: \"kubernetes.io/projected/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-kube-api-access-9svr9\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls\" (UID: \"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"
Oct 30 19:00:00.316949 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.316820    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb6c9b7-0581-42e2-9632-144ee021879b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72\" (UID: \"ddb6c9b7-0581-42e2-9632-144ee021879b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"
Oct 30 19:00:00.316949 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.316923    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7plc\" (UniqueName: \"kubernetes.io/projected/ddb6c9b7-0581-42e2-9632-144ee021879b-kube-api-access-g7plc\") pod \"maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72\" (UID: \"ddb6c9b7-0581-42e2-9632-144ee021879b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"
Oct 30 19:00:00.317121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.316965    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsp7r\" (UniqueName: \"kubernetes.io/projected/19b5b0a1-ea85-49d5-96a1-b417620ffacb-kube-api-access-wsp7r\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5\" (UID: \"19b5b0a1-ea85-49d5-96a1-b417620ffacb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"
Oct 30 19:00:00.317121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.317040    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b5b0a1-ea85-49d5-96a1-b417620ffacb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5\" (UID: \"19b5b0a1-ea85-49d5-96a1-b417620ffacb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"
Oct 30 19:00:00.317833 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.317809    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b5b0a1-ea85-49d5-96a1-b417620ffacb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5\" (UID: \"19b5b0a1-ea85-49d5-96a1-b417620ffacb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"
Oct 30 19:00:00.317971 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.316215    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls\" (UID: \"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"
Oct 30 19:00:00.318071 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.315065    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw\" (UID: \"dd2f7f58-73ee-4675-9d18-b6d8595e2af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"
Oct 30 19:00:00.319110 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.319090    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"]
Oct 30 19:00:00.374318 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.374283    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsp7r\" (UniqueName: \"kubernetes.io/projected/19b5b0a1-ea85-49d5-96a1-b417620ffacb-kube-api-access-wsp7r\") pod \"maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5\" (UID: \"19b5b0a1-ea85-49d5-96a1-b417620ffacb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"
Oct 30 19:00:00.374660 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.374641    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9svr9\" (UniqueName: \"kubernetes.io/projected/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-kube-api-access-9svr9\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls\" (UID: \"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"
Oct 30 19:00:00.374874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.374857    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlkbn\" (UniqueName: \"kubernetes.io/projected/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-kube-api-access-qlkbn\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw\" (UID: \"dd2f7f58-73ee-4675-9d18-b6d8595e2af5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"
Oct 30 19:00:00.418707 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.418627    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb6c9b7-0581-42e2-9632-144ee021879b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72\" (UID: \"ddb6c9b7-0581-42e2-9632-144ee021879b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"
Oct 30 19:00:00.418862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.418767    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7plc\" (UniqueName: \"kubernetes.io/projected/ddb6c9b7-0581-42e2-9632-144ee021879b-kube-api-access-g7plc\") pod \"maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72\" (UID: \"ddb6c9b7-0581-42e2-9632-144ee021879b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"
Oct 30 19:00:00.419259 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.419238    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb6c9b7-0581-42e2-9632-144ee021879b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72\" (UID: \"ddb6c9b7-0581-42e2-9632-144ee021879b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"
Oct 30 19:00:00.477083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.477045    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7plc\" (UniqueName: \"kubernetes.io/projected/ddb6c9b7-0581-42e2-9632-144ee021879b-kube-api-access-g7plc\") pod \"maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72\" (UID: \"ddb6c9b7-0581-42e2-9632-144ee021879b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"
Oct 30 19:00:00.485762 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.485735    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"
Oct 30 19:00:00.493075 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.493055    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"
Oct 30 19:00:00.502868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.502834    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"
Oct 30 19:00:00.530520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.530469    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"
Oct 30 19:00:00.875157 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:00:00.875128    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fa5b52_0e16_41bb_8c5c_1e78fe85beb9.slice/crio-66727a1e8c1ef6177ac41759d61d4701c78a2cc70db27cf1e9fbaaa1aa02c270 WatchSource:0}: Error finding container 66727a1e8c1ef6177ac41759d61d4701c78a2cc70db27cf1e9fbaaa1aa02c270: Status 404 returned error can't find the container with id 66727a1e8c1ef6177ac41759d61d4701c78a2cc70db27cf1e9fbaaa1aa02c270
Oct 30 19:00:00.875663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.875576    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"]
Oct 30 19:00:00.925095 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:00:00.925000    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b5b0a1_ea85_49d5_96a1_b417620ffacb.slice/crio-b5876e38b7face50f615a7d588f78df121319cdadd2ec5195c8659fc9988f446 WatchSource:0}: Error finding container b5876e38b7face50f615a7d588f78df121319cdadd2ec5195c8659fc9988f446: Status 404 returned error can't find the container with id b5876e38b7face50f615a7d588f78df121319cdadd2ec5195c8659fc9988f446
Oct 30 19:00:00.926479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.926455    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"]
Oct 30 19:00:00.927633 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:00:00.927612    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2f7f58_73ee_4675_9d18_b6d8595e2af5.slice/crio-b7e082ecb39311dc156f9293523cf8b288d37281fb74816491dc69287d728612 WatchSource:0}: Error finding container b7e082ecb39311dc156f9293523cf8b288d37281fb74816491dc69287d728612: Status 404 returned error can't find the container with id b7e082ecb39311dc156f9293523cf8b288d37281fb74816491dc69287d728612
Oct 30 19:00:00.930571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.930550    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"]
Oct 30 19:00:00.960675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:00.960649    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"]
Oct 30 19:00:01.383647 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:01.383558    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" event={"ID":"ddb6c9b7-0581-42e2-9632-144ee021879b","Type":"ContainerStarted","Data":"515ac1767f39e41ee3c0b97d6165f891407bbbf35aaa44279d41181cc58f991a"}
Oct 30 19:00:01.383647 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:01.383621    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" event={"ID":"ddb6c9b7-0581-42e2-9632-144ee021879b","Type":"ContainerStarted","Data":"446086258a59083293d615f2f73cde4210534236f913dadc747b4e360ad6fb94"}
Oct 30 19:00:01.385364 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:01.385327    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" event={"ID":"19b5b0a1-ea85-49d5-96a1-b417620ffacb","Type":"ContainerStarted","Data":"b38f51b2d97aa468fc90e8c0fa49600b0de3c183aa3a06364fcdc7890c02061d"}
Oct 30 19:00:01.385478 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:01.385381    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" event={"ID":"19b5b0a1-ea85-49d5-96a1-b417620ffacb","Type":"ContainerStarted","Data":"b5876e38b7face50f615a7d588f78df121319cdadd2ec5195c8659fc9988f446"}
Oct 30 19:00:01.386804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:01.386780    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw" event={"ID":"dd2f7f58-73ee-4675-9d18-b6d8595e2af5","Type":"ContainerStarted","Data":"b7e082ecb39311dc156f9293523cf8b288d37281fb74816491dc69287d728612"}
Oct 30 19:00:01.388592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:01.388566    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" event={"ID":"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9","Type":"ContainerStarted","Data":"fc95c84c204d8cd96fec641a1cc842e5e07f15be4f6de62ddc28f5ca2ba6f73b"}
Oct 30 19:00:01.388672 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:01.388614    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" event={"ID":"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9","Type":"ContainerStarted","Data":"66727a1e8c1ef6177ac41759d61d4701c78a2cc70db27cf1e9fbaaa1aa02c270"}
Oct 30 19:00:01.499173 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:01.499117    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" podStartSLOduration=1.499101776 podStartE2EDuration="1.499101776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:01.495397767 +0000 UTC m=+1292.383314859" watchObservedRunningTime="2026-10-30 19:00:01.499101776 +0000 UTC m=+1292.387018862"
Oct 30 19:00:01.613205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:01.613139    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" podStartSLOduration=1.61312051 podStartE2EDuration="1.61312051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:01.61129576 +0000 UTC m=+1292.499212850" watchObservedRunningTime="2026-10-30 19:00:01.61312051 +0000 UTC m=+1292.501037600"
Oct 30 19:00:01.727203 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:01.727142    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" podStartSLOduration=1.7271281840000001 podStartE2EDuration="1.727128184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:01.724113482 +0000 UTC m=+1292.612030571" watchObservedRunningTime="2026-10-30 19:00:01.727128184 +0000 UTC m=+1292.615045274"
Oct 30 19:00:02.397441 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.397356    4267 generic.go:356] "Generic (PLEG): container finished" podID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerID="defa26b21ab8c936fbbdd5b1401adecc82d2e70b3416b3468d813e13be1dfed5" exitCode=60
Oct 30 19:00:02.397884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.397447    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw" event={"ID":"dd2f7f58-73ee-4675-9d18-b6d8595e2af5","Type":"ContainerDied","Data":"defa26b21ab8c936fbbdd5b1401adecc82d2e70b3416b3468d813e13be1dfed5"}
Oct 30 19:00:02.397884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.397728    4267 scope.go:122] "RemoveContainer" containerID="defa26b21ab8c936fbbdd5b1401adecc82d2e70b3416b3468d813e13be1dfed5"
Oct 30 19:00:02.399188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.399162    4267 generic.go:356] "Generic (PLEG): container finished" podID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerID="fc95c84c204d8cd96fec641a1cc842e5e07f15be4f6de62ddc28f5ca2ba6f73b" exitCode=60
Oct 30 19:00:02.399295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.399222    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" event={"ID":"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9","Type":"ContainerDied","Data":"fc95c84c204d8cd96fec641a1cc842e5e07f15be4f6de62ddc28f5ca2ba6f73b"}
Oct 30 19:00:02.399571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.399551    4267 scope.go:122] "RemoveContainer" containerID="fc95c84c204d8cd96fec641a1cc842e5e07f15be4f6de62ddc28f5ca2ba6f73b"
Oct 30 19:00:02.400973 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.400933    4267 generic.go:356] "Generic (PLEG): container finished" podID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerID="515ac1767f39e41ee3c0b97d6165f891407bbbf35aaa44279d41181cc58f991a" exitCode=60
Oct 30 19:00:02.401042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.401018    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" event={"ID":"ddb6c9b7-0581-42e2-9632-144ee021879b","Type":"ContainerDied","Data":"515ac1767f39e41ee3c0b97d6165f891407bbbf35aaa44279d41181cc58f991a"}
Oct 30 19:00:02.401280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.401262    4267 scope.go:122] "RemoveContainer" containerID="515ac1767f39e41ee3c0b97d6165f891407bbbf35aaa44279d41181cc58f991a"
Oct 30 19:00:02.402878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.402857    4267 generic.go:356] "Generic (PLEG): container finished" podID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerID="b38f51b2d97aa468fc90e8c0fa49600b0de3c183aa3a06364fcdc7890c02061d" exitCode=60
Oct 30 19:00:02.402965 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.402934    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" event={"ID":"19b5b0a1-ea85-49d5-96a1-b417620ffacb","Type":"ContainerDied","Data":"b38f51b2d97aa468fc90e8c0fa49600b0de3c183aa3a06364fcdc7890c02061d"}
Oct 30 19:00:02.403174 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:02.403158    4267 scope.go:122] "RemoveContainer" containerID="b38f51b2d97aa468fc90e8c0fa49600b0de3c183aa3a06364fcdc7890c02061d"
Oct 30 19:00:03.413912 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.413877    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" event={"ID":"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9","Type":"ContainerStarted","Data":"ad4c2183023bb8b77134ff2b08a50e33b4ff5167c2ecb49308fba654e961ac53"}
Oct 30 19:00:03.416224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.416192    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" event={"ID":"ddb6c9b7-0581-42e2-9632-144ee021879b","Type":"ContainerStarted","Data":"3c5d93ac1c8396041f4b4de081fd2fd978a2771e8d7dea121980ef1d6dd2f964"}
Oct 30 19:00:03.419050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.419024    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" event={"ID":"19b5b0a1-ea85-49d5-96a1-b417620ffacb","Type":"ContainerStarted","Data":"b1143313d92c08f41ea204cc8b49375c7eab7bbed94cf4767687acd58743de94"}
Oct 30 19:00:03.419280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.419263    4267 scope.go:122] "RemoveContainer" containerID="b1143313d92c08f41ea204cc8b49375c7eab7bbed94cf4767687acd58743de94"
Oct 30 19:00:03.419600 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:03.419575    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5_odh-ai-gateway-infra(19b5b0a1-ea85-49d5-96a1-b417620ffacb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb"
Oct 30 19:00:03.421012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.420984    4267 generic.go:356] "Generic (PLEG): container finished" podID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerID="34daa1b796c79387aada0157db0b281ef6313b8a35fd95f412f10ebc0ee9f037" exitCode=60
Oct 30 19:00:03.421110 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.421063    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw" event={"ID":"dd2f7f58-73ee-4675-9d18-b6d8595e2af5","Type":"ContainerDied","Data":"34daa1b796c79387aada0157db0b281ef6313b8a35fd95f412f10ebc0ee9f037"}
Oct 30 19:00:03.421169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.421127    4267 scope.go:122] "RemoveContainer" containerID="defa26b21ab8c936fbbdd5b1401adecc82d2e70b3416b3468d813e13be1dfed5"
Oct 30 19:00:03.421337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.421318    4267 scope.go:122] "RemoveContainer" containerID="34daa1b796c79387aada0157db0b281ef6313b8a35fd95f412f10ebc0ee9f037"
Oct 30 19:00:03.421663 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:03.421639    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw_odh-ai-gateway-infra(dd2f7f58-73ee-4675-9d18-b6d8595e2af5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5"
Oct 30 19:00:03.531727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.531690    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"]
Oct 30 19:00:03.547084 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.547043    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.563293 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.563206    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"]
Oct 30 19:00:03.567400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.567370    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.599100 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.599064    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"]
Oct 30 19:00:03.646469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.646435    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"]
Oct 30 19:00:03.658719 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.658680    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.658719 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.658723    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.658966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.658816    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.658966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.658884    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6981301a-339d-4827-b951-cf81a8e00d2f-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.658966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.658911    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.658966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.658935    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.659196 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.658973    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.659196 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659024    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7qtw\" (UniqueName: \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-kube-api-access-t7qtw\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.659196 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659042    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.659196 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659109    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.659196 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659128    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.659196 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659196    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.659608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659227    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6981301a-339d-4827-b951-cf81a8e00d2f-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.659608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659245    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.659608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659324    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.659608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659410    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.659608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659443    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpvlc\" (UniqueName: \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-kube-api-access-mpvlc\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.659608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.659484    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.760453 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.760408    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6981301a-339d-4827-b951-cf81a8e00d2f-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.760453 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.760449    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.760704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.760515    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.760704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.760554    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.760704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.760590    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvlc\" (UniqueName: \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-kube-api-access-mpvlc\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.760704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.760614    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.760704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.760690    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.760995 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.760965    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.761086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761066    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.761144 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761103    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.761309 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761287    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.761370 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761109    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6981301a-339d-4827-b951-cf81a8e00d2f-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.761370 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761361    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.761471 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761368    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.761471 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761388    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.761471 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761390    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.761471 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761434    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.761471 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761440    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.761724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761531    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7qtw\" (UniqueName: \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-kube-api-access-t7qtw\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.761724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761565    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.761724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761602    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.761724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761627    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.761724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761683    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-certs\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.761724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.761705    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.762045 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.762020    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6981301a-339d-4827-b951-cf81a8e00d2f-istiod-ca-cert\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.762364 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.762341    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-credential-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.762854 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.762823    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-socket\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.762990 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.762860    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-data\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.763630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.763603    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.764007 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.763984    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6981301a-339d-4827-b951-cf81a8e00d2f-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.764151 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.764135    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-envoy\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.764610 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.764588    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-podinfo\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.819467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.819380    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7qtw\" (UniqueName: \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-kube-api-access-t7qtw\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.819467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.819380    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.819885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.819863    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-token\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.820019 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.819998    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpvlc\" (UniqueName: \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-kube-api-access-mpvlc\") pod \"e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") " pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:03.884058 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.884024    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:03.891300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:03.891276    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:04.320187 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:00:04.320161    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6981301a_339d_4827_b951_cf81a8e00d2f.slice/crio-64bedd35e1cb8b1dd82684afb211e62dc87ddd324fabc0884660f0ea583c2703 WatchSource:0}: Error finding container 64bedd35e1cb8b1dd82684afb211e62dc87ddd324fabc0884660f0ea583c2703: Status 404 returned error can't find the container with id 64bedd35e1cb8b1dd82684afb211e62dc87ddd324fabc0884660f0ea583c2703
Oct 30 19:00:04.321083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.321055    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"]
Oct 30 19:00:04.323275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.323236    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:04.323360 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:00:04.323312    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1e43fc_08d6_4ea7_8065_14a0d0b77c36.slice/crio-aba7f506fb76b9a377f0495aa38023880f13dede9c613f51d9c45219ccc42254 WatchSource:0}: Error finding container aba7f506fb76b9a377f0495aa38023880f13dede9c613f51d9c45219ccc42254: Status 404 returned error can't find the container with id aba7f506fb76b9a377f0495aa38023880f13dede9c613f51d9c45219ccc42254
Oct 30 19:00:04.323360 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.323322    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:04.323447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.323394    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:04.325761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.325726    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"]
Oct 30 19:00:04.325898 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.325877    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:04.325942 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.325929    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:04.325974 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.325959    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:04.435545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.434708    4267 scope.go:122] "RemoveContainer" containerID="34daa1b796c79387aada0157db0b281ef6313b8a35fd95f412f10ebc0ee9f037"
Oct 30 19:00:04.435545 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:04.435052    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw_odh-ai-gateway-infra(dd2f7f58-73ee-4675-9d18-b6d8595e2af5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5"
Oct 30 19:00:04.436508 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.436465    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" event={"ID":"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36","Type":"ContainerStarted","Data":"0cece4f63bb67fa2e815ef7c5ed7a131215b6891cfa93259e35e9d2ed6d5d805"}
Oct 30 19:00:04.436599 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.436569    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" event={"ID":"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36","Type":"ContainerStarted","Data":"aba7f506fb76b9a377f0495aa38023880f13dede9c613f51d9c45219ccc42254"}
Oct 30 19:00:04.438845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.438819    4267 generic.go:356] "Generic (PLEG): container finished" podID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerID="ad4c2183023bb8b77134ff2b08a50e33b4ff5167c2ecb49308fba654e961ac53" exitCode=60
Oct 30 19:00:04.438929 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.438916    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" event={"ID":"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9","Type":"ContainerDied","Data":"ad4c2183023bb8b77134ff2b08a50e33b4ff5167c2ecb49308fba654e961ac53"}
Oct 30 19:00:04.438989 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.438962    4267 scope.go:122] "RemoveContainer" containerID="fc95c84c204d8cd96fec641a1cc842e5e07f15be4f6de62ddc28f5ca2ba6f73b"
Oct 30 19:00:04.439389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.439371    4267 scope.go:122] "RemoveContainer" containerID="ad4c2183023bb8b77134ff2b08a50e33b4ff5167c2ecb49308fba654e961ac53"
Oct 30 19:00:04.439736 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:04.439705    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls_odh-ai-gateway-infra(34fa5b52-0e16-41bb-8c5c-1e78fe85beb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9"
Oct 30 19:00:04.443051 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.443024    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" event={"ID":"6981301a-339d-4827-b951-cf81a8e00d2f","Type":"ContainerStarted","Data":"e9eef03fed5643b81b835a76483bdb1a5e2114d6e925da827b6c2dd07a1578ed"}
Oct 30 19:00:04.443125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.443078    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" event={"ID":"6981301a-339d-4827-b951-cf81a8e00d2f","Type":"ContainerStarted","Data":"64bedd35e1cb8b1dd82684afb211e62dc87ddd324fabc0884660f0ea583c2703"}
Oct 30 19:00:04.446545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.446487    4267 generic.go:356] "Generic (PLEG): container finished" podID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerID="3c5d93ac1c8396041f4b4de081fd2fd978a2771e8d7dea121980ef1d6dd2f964" exitCode=60
Oct 30 19:00:04.446696 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.446617    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" event={"ID":"ddb6c9b7-0581-42e2-9632-144ee021879b","Type":"ContainerDied","Data":"3c5d93ac1c8396041f4b4de081fd2fd978a2771e8d7dea121980ef1d6dd2f964"}
Oct 30 19:00:04.446972 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.446953    4267 scope.go:122] "RemoveContainer" containerID="3c5d93ac1c8396041f4b4de081fd2fd978a2771e8d7dea121980ef1d6dd2f964"
Oct 30 19:00:04.447257 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:04.447234    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72_odh-ai-gateway-infra(ddb6c9b7-0581-42e2-9632-144ee021879b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b"
Oct 30 19:00:04.449560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.449536    4267 generic.go:356] "Generic (PLEG): container finished" podID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerID="b1143313d92c08f41ea204cc8b49375c7eab7bbed94cf4767687acd58743de94" exitCode=60
Oct 30 19:00:04.449660 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.449595    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" event={"ID":"19b5b0a1-ea85-49d5-96a1-b417620ffacb","Type":"ContainerDied","Data":"b1143313d92c08f41ea204cc8b49375c7eab7bbed94cf4767687acd58743de94"}
Oct 30 19:00:04.449930 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.449913    4267 scope.go:122] "RemoveContainer" containerID="b1143313d92c08f41ea204cc8b49375c7eab7bbed94cf4767687acd58743de94"
Oct 30 19:00:04.450180 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:04.450158    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5_odh-ai-gateway-infra(19b5b0a1-ea85-49d5-96a1-b417620ffacb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb"
Oct 30 19:00:04.460685 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.460654    4267 scope.go:122] "RemoveContainer" containerID="515ac1767f39e41ee3c0b97d6165f891407bbbf35aaa44279d41181cc58f991a"
Oct 30 19:00:04.495720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.495586    4267 scope.go:122] "RemoveContainer" containerID="b38f51b2d97aa468fc90e8c0fa49600b0de3c183aa3a06364fcdc7890c02061d"
Oct 30 19:00:04.868531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.868478    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"]
Oct 30 19:00:04.873695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.873675    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:04.884932 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.884907    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:04.892289 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.892260    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:04.895772 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.895718    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podStartSLOduration=1.895703926 podStartE2EDuration="1.895703926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:04.893797769 +0000 UTC m=+1295.781714860" watchObservedRunningTime="2026-10-30 19:00:04.895703926 +0000 UTC m=+1295.783621092"
Oct 30 19:00:04.950036 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.949999    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"]
Oct 30 19:00:04.977961 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.977922    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:04.978185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.978030    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wsr8\" (UniqueName: \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-kube-api-access-4wsr8\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:04.978185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.978077    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:04.978263 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.978192    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:04.978263 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.978234    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:04.978329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.978286    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65fda7a-34ea-4733-ae35-22813f1654b4-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:04.978329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.978311    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:04.978429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.978402    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:04.978463 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:04.978455    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.036655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.036599    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podStartSLOduration=2.036584211 podStartE2EDuration="2.036584211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:05.035472105 +0000 UTC m=+1295.923389196" watchObservedRunningTime="2026-10-30 19:00:05.036584211 +0000 UTC m=+1295.924501300"
Oct 30 19:00:05.080092 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080049    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.080269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080118    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wsr8\" (UniqueName: \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-kube-api-access-4wsr8\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.080269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080159    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.080269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080246    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.080450 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080275    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.080450 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080314    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65fda7a-34ea-4733-ae35-22813f1654b4-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.080450 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080357    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.080450 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080445    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.080663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080476    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.080755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080727    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.080907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080874    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.081037 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.080947    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.081037 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.081014    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.081160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.081136    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65fda7a-34ea-4733-ae35-22813f1654b4-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.082946 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.082926    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.083689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.083665    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.148583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.148472    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"]
Oct 30 19:00:05.150509 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.150457    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.153887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.153456    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wsr8\" (UniqueName: \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-kube-api-access-4wsr8\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.165190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.165152    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.181937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.181865    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.182180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.181982    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.182180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.182025    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdzjc\" (UniqueName: \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-kube-api-access-bdzjc\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.182180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.182061    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.182180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.182148    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.182374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.182208    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.182374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.182243    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.182374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.182360    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.182479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.182387    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75951caa-ff7b-4c5b-b55b-11a66a055f17-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.188478 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.188452    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:05.284715 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.284461    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.300620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.299643    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.300620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.299809    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.300620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.299845    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75951caa-ff7b-4c5b-b55b-11a66a055f17-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.300620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.299949    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.300620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.300017    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.300620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.300043    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzjc\" (UniqueName: \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-kube-api-access-bdzjc\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.300620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.300086    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.300620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.300115    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.300620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.300233    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.301355 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.301324    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.301648 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.301623    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.303168 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.302312    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75951caa-ff7b-4c5b-b55b-11a66a055f17-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.303168 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.302900    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.303912 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.303889    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.306741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.306713    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.334522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.315849    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"]
Oct 30 19:00:05.368989 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.368926    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzjc\" (UniqueName: \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-kube-api-access-bdzjc\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.369641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.369618    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.499621 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.499596    4267 scope.go:122] "RemoveContainer" containerID="ad4c2183023bb8b77134ff2b08a50e33b4ff5167c2ecb49308fba654e961ac53"
Oct 30 19:00:05.499990 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:05.499892    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls_odh-ai-gateway-infra(34fa5b52-0e16-41bb-8c5c-1e78fe85beb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9"
Oct 30 19:00:05.503283 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.503251    4267 scope.go:122] "RemoveContainer" containerID="3c5d93ac1c8396041f4b4de081fd2fd978a2771e8d7dea121980ef1d6dd2f964"
Oct 30 19:00:05.503548 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:05.503505    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72_odh-ai-gateway-infra(ddb6c9b7-0581-42e2-9632-144ee021879b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b"
Oct 30 19:00:05.505713 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.505694    4267 scope.go:122] "RemoveContainer" containerID="b1143313d92c08f41ea204cc8b49375c7eab7bbed94cf4767687acd58743de94"
Oct 30 19:00:05.506301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.506112    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:05.506714 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:05.506685    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5_odh-ai-gateway-infra(19b5b0a1-ea85-49d5-96a1-b417620ffacb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb"
Oct 30 19:00:05.858595 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.817599    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"]
Oct 30 19:00:05.884006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.883085    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:05.884006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.883186    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:05.884006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.883233    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:05.887773 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.887056    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:00:05.887773 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.887107    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:00:05.892766 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.892738    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:00:05.892890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:05.892790    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:00:06.156116 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.155359    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:06.156116 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.155464    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:06.156116 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.155530    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:06.171924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.169676    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"]
Oct 30 19:00:06.594099 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.592953    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" event={"ID":"75951caa-ff7b-4c5b-b55b-11a66a055f17","Type":"ContainerStarted","Data":"af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d"}
Oct 30 19:00:06.594099 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.593027    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" event={"ID":"75951caa-ff7b-4c5b-b55b-11a66a055f17","Type":"ContainerStarted","Data":"f17b78de18ad575c55f9ac3342e9f8b31467f89014f7270a4730d0218740c98e"}
Oct 30 19:00:06.598066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.598035    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" event={"ID":"b65fda7a-34ea-4733-ae35-22813f1654b4","Type":"ContainerStarted","Data":"cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1"}
Oct 30 19:00:06.598295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.598274    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" event={"ID":"b65fda7a-34ea-4733-ae35-22813f1654b4","Type":"ContainerStarted","Data":"65c55fca4faf02ad2d045f7669a2a37e703d4e05570e9167dda97897645bd681"}
Oct 30 19:00:06.861830 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.861703    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podStartSLOduration=2.861677864 podStartE2EDuration="2.861677864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:06.851046384 +0000 UTC m=+1297.738963475" watchObservedRunningTime="2026-10-30 19:00:06.861677864 +0000 UTC m=+1297.749594955"
Oct 30 19:00:06.870296 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.870232    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podStartSLOduration=1.8702160490000002 podStartE2EDuration="1.870216049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:06.72860302 +0000 UTC m=+1297.616520110" watchObservedRunningTime="2026-10-30 19:00:06.870216049 +0000 UTC m=+1297.758133140"
Oct 30 19:00:06.884976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.884939    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:00:06.885215 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.885186    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:00:06.891848 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.891818    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:00:06.892028 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:06.892005    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:00:07.189797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:07.189770    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:07.191045 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:07.191013    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:00:07.191157 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:07.191072    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:00:07.508584 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:07.508359    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:07.508584 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:07.508515    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:00:07.508817 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:07.508580    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:00:07.888572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:07.888093    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:00:07.888572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:07.888213    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:00:07.901521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:07.892503    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:00:07.901521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:07.892577    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:00:08.192521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:08.191875    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:00:08.192521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:08.191935    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:00:08.518575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:08.517895    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:00:08.518575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:08.517963    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:00:08.868574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:08.867602    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:00:08.889374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:08.889334    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:00:08.890059 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:08.890026    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:00:08.904730 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:08.904609    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:00:08.904730 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:08.904673    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:00:08.979626 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:08.979578    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:09.115517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.099683    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:09.182574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.179608    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:09.189217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.189186    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:00:09.189443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.189418    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:00:09.287340 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.287292    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:09.313230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.313154    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:09.331645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.331568    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:09.479517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.478569    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:09.507580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.506853    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:00:09.507580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.506914    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:00:09.875698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.875594    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:00:09.877363 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.877336    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:00:09.918161 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.918129    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:00:09.918750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.918720    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:00:09.918914 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.918897    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:00:09.919029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.919012    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:00:09.997518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.981579    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:09.997518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:09.982072    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:00:10.016524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.006597    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:10.218579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.168946    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 30 19:00:10.218579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.197359    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:00:10.218579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.197413    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:00:10.272934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.219996    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:10.272934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.248611    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:10.350068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.332444    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:10.350068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.348660    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:10.488575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.479961    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:10.526528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.526318    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:00:10.526528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.526384    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:00:10.537531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.535645    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:00:10.537531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.535724    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:00:10.537531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.535762    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:00:10.683516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.682619    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:00:10.980581 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.980124    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded"
Oct 30 19:00:10.980581 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:10.980507    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:11.028518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.028299    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:11.036521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.031076    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.068677 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.068443    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.068677 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.068547    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:00:11.111142 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.108693    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:11.111142 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.110310    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:00:11.116206 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.116171    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:00:11.171562 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.156072    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:00:11.171562 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.157609    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:00:11.296904 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.296810    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.297166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.297134    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:00:11.337566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.329439    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:00:11.338393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.338335    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:11.340119 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.340090    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:00:11.510520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.508941    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.510520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.509003    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:00:11.550644 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.550528    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:11.567516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.557630    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:11.571724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.571695    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:00:11.583660 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.583634    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:00:11.718522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.714606    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:11.718522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.715060    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:00:11.890584 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.889725    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.890584 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.889801    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:00:11.895521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.892202    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:00:11.895521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:11.892252    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:00:12.087517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.077606    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.087517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.079560    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.087517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.079893    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 30 19:00:12.087517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.081592    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.116529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.110103    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.163785 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.163645    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:00:12.164262 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.164240    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:00:12.190540 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.170545    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.190540 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.171390    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:00:12.206549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.206131    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:00:12.206549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.206198    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:00:12.348518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.347591    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:00:12.348518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.347990    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:00:12.479516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.474537    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.512250 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.512213    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:00:12.512528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.512480    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:00:12.596519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.559624    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.596519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.560975    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.684518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.668780    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:00:12.801639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.796547    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:00:12.803729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.803667    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.905957 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.905919    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:00:12.919716 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.913876    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 19:00:12.919716 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.919659    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 19:00:12.928586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:12.928536    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:00:13.084583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.082610    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.092518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.092074    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.129512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.128480    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:00:13.138202 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.131114    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:00:13.156535 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.156485    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:00:13.177851 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.166756    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.177851 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.177356    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.194906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.194601    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:00:13.194906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.194676    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:00:13.348665 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.348563    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:00:13.388734 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.349624    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.490697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.490642    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.526917 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.526882    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:00:13.527201 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.527170    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:00:13.809534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.803580    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.974605 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:13.974558    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"]
Oct 30 19:00:14.093437 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.093329    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.094831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.094696    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:00:14.128642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.128604    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"]
Oct 30 19:00:14.131776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.131712    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.211524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.171161    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.211524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.177599    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.241518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.217294    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-975cf5\"/\"distinct-w1-975cf5-kserve-self-signed-certs\""
Oct 30 19:00:14.241518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.228619    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:00:14.241518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.228692    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:00:14.333519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.332639    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-tmp-dir\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.333519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.332724    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b650c63a-b609-4a9c-83df-6dd942456b5d-tls-certs\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.333519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.332790    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-dshm\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.333519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.332869    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk85b\" (UniqueName: \"kubernetes.io/projected/b650c63a-b609-4a9c-83df-6dd942456b5d-kube-api-access-mk85b\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.333519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.332988    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-model-cache\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.333519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.333070    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-home\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.379572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.365110    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:00:14.379572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.367477    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.437520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.426530    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:00:14.437520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.426751    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:00:14.447795 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.447741    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:00:14.497193 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.497158    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b650c63a-b609-4a9c-83df-6dd942456b5d-tls-certs\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.503533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.503481    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-dshm\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.504797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.504772    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk85b\" (UniqueName: \"kubernetes.io/projected/b650c63a-b609-4a9c-83df-6dd942456b5d-kube-api-access-mk85b\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.512802 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.512766    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-model-cache\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.514741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.514718    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-home\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.515063 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.515040    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-tmp-dir\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.515805 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.515755    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-tmp-dir\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.516879 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.516857    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-model-cache\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.580463 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.502691    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.581554 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.519914    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-home\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.581750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.554207    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:00:14.581910 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.581880    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:00:14.582038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.567657    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.606560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.606366    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk85b\" (UniqueName: \"kubernetes.io/projected/b650c63a-b609-4a9c-83df-6dd942456b5d-kube-api-access-mk85b\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.611526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.609295    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b650c63a-b609-4a9c-83df-6dd942456b5d-tls-certs\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.855945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.821670    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:00:14.855945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.827934    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:14.855945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.829941    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:14.855945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.832706    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.855945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.834420    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:00:14.878786 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.878754    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-dshm\") pod \"distinct-w1-975cf5-kserve-8f54c54b5-m6msj\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") " pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:14.909974 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.908089    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:00:14.959567 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.958233    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:00:14.959567 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.958315    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:00:14.983516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.977754    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:00:14.983516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.977854    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:14.983516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.980657    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:14.994637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.994117    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:00:14.996387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:14.996363    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:00:15.108269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:15.105335    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:15.189424 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:15.189384    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:00:15.189645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:15.189446    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:00:15.507533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:15.507479    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:00:15.507812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:15.507784    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:00:15.606972 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:00:15.606920    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb650c63a_b609_4a9c_83df_6dd942456b5d.slice/crio-7229d07686ec29cde00336b8536f295848e008ed86d9140d3a4f3908bdb271d8 WatchSource:0}: Error finding container 7229d07686ec29cde00336b8536f295848e008ed86d9140d3a4f3908bdb271d8: Status 404 returned error can't find the container with id 7229d07686ec29cde00336b8536f295848e008ed86d9140d3a4f3908bdb271d8
Oct 30 19:00:15.607317 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:15.607289    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"]
Oct 30 19:00:15.875581 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:15.874726    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" event={"ID":"b650c63a-b609-4a9c-83df-6dd942456b5d","Type":"ContainerStarted","Data":"daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13"}
Oct 30 19:00:15.875581 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:15.874793    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" event={"ID":"b650c63a-b609-4a9c-83df-6dd942456b5d","Type":"ContainerStarted","Data":"7229d07686ec29cde00336b8536f295848e008ed86d9140d3a4f3908bdb271d8"}
Oct 30 19:00:16.095559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:16.094164    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podStartSLOduration=3.094143115 podStartE2EDuration="3.094143115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:16.092962941 +0000 UTC m=+1306.980880032" watchObservedRunningTime="2026-10-30 19:00:16.094143115 +0000 UTC m=+1306.982060205"
Oct 30 19:00:16.105731 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:16.105685    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:16.458299 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:16.458263    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:16.459430 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:16.459396    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:16.460858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:16.460831    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:00:16.571760 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:16.571727    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:16.572185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:16.572165    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:16.572969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:16.572951    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:00:17.214327 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:17.214288    4267 scope.go:122] "RemoveContainer" containerID="ad4c2183023bb8b77134ff2b08a50e33b4ff5167c2ecb49308fba654e961ac53"
Oct 30 19:00:17.214563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:17.214424    4267 scope.go:122] "RemoveContainer" containerID="34daa1b796c79387aada0157db0b281ef6313b8a35fd95f412f10ebc0ee9f037"
Oct 30 19:00:17.214563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:17.214527    4267 scope.go:122] "RemoveContainer" containerID="b1143313d92c08f41ea204cc8b49375c7eab7bbed94cf4767687acd58743de94"
Oct 30 19:00:17.901765 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:17.901670    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw" event={"ID":"dd2f7f58-73ee-4675-9d18-b6d8595e2af5","Type":"ContainerStarted","Data":"e97e42bffc596f1c7d054158ec6c2ebf90a00c1ab359e63201a31537c101a40e"}
Oct 30 19:00:17.903911 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:17.903886    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" event={"ID":"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9","Type":"ContainerStarted","Data":"b35c3a94a8c677bdecd1905710a7ecb4ddc65e0183dc5dc17a51a908e8cd795e"}
Oct 30 19:00:17.906208 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:17.906183    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" event={"ID":"19b5b0a1-ea85-49d5-96a1-b417620ffacb","Type":"ContainerStarted","Data":"e3e857d768fed2aef7fdc922c02e51dc79cdcae4c4d44621f5ed06588c56ff4b"}
Oct 30 19:00:18.016849 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.016796    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw" podStartSLOduration=18.016782377 podStartE2EDuration="18.016782377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:18.013875093 +0000 UTC m=+1308.901792184" watchObservedRunningTime="2026-10-30 19:00:18.016782377 +0000 UTC m=+1308.904699519"
Oct 30 19:00:18.213614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.213582    4267 scope.go:122] "RemoveContainer" containerID="3c5d93ac1c8396041f4b4de081fd2fd978a2771e8d7dea121980ef1d6dd2f964"
Oct 30 19:00:18.916074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.915989    4267 generic.go:356] "Generic (PLEG): container finished" podID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerID="e3e857d768fed2aef7fdc922c02e51dc79cdcae4c4d44621f5ed06588c56ff4b" exitCode=60
Oct 30 19:00:18.916546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.916061    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" event={"ID":"19b5b0a1-ea85-49d5-96a1-b417620ffacb","Type":"ContainerDied","Data":"e3e857d768fed2aef7fdc922c02e51dc79cdcae4c4d44621f5ed06588c56ff4b"}
Oct 30 19:00:18.916546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.916134    4267 scope.go:122] "RemoveContainer" containerID="b1143313d92c08f41ea204cc8b49375c7eab7bbed94cf4767687acd58743de94"
Oct 30 19:00:18.916546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.916532    4267 scope.go:122] "RemoveContainer" containerID="e3e857d768fed2aef7fdc922c02e51dc79cdcae4c4d44621f5ed06588c56ff4b"
Oct 30 19:00:18.916903 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:18.916877    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5_odh-ai-gateway-infra(19b5b0a1-ea85-49d5-96a1-b417620ffacb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb"
Oct 30 19:00:18.918152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.918131    4267 generic.go:356] "Generic (PLEG): container finished" podID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerID="e97e42bffc596f1c7d054158ec6c2ebf90a00c1ab359e63201a31537c101a40e" exitCode=60
Oct 30 19:00:18.918278 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.918199    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw" event={"ID":"dd2f7f58-73ee-4675-9d18-b6d8595e2af5","Type":"ContainerDied","Data":"e97e42bffc596f1c7d054158ec6c2ebf90a00c1ab359e63201a31537c101a40e"}
Oct 30 19:00:18.918589 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.918574    4267 scope.go:122] "RemoveContainer" containerID="e97e42bffc596f1c7d054158ec6c2ebf90a00c1ab359e63201a31537c101a40e"
Oct 30 19:00:18.919739 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:18.919708    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw_odh-ai-gateway-infra(dd2f7f58-73ee-4675-9d18-b6d8595e2af5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5"
Oct 30 19:00:18.921090 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.921061    4267 generic.go:356] "Generic (PLEG): container finished" podID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerID="b35c3a94a8c677bdecd1905710a7ecb4ddc65e0183dc5dc17a51a908e8cd795e" exitCode=60
Oct 30 19:00:18.921298 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.921277    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" event={"ID":"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9","Type":"ContainerDied","Data":"b35c3a94a8c677bdecd1905710a7ecb4ddc65e0183dc5dc17a51a908e8cd795e"}
Oct 30 19:00:18.921827 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.921804    4267 scope.go:122] "RemoveContainer" containerID="b35c3a94a8c677bdecd1905710a7ecb4ddc65e0183dc5dc17a51a908e8cd795e"
Oct 30 19:00:18.922326 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:18.922299    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls_odh-ai-gateway-infra(34fa5b52-0e16-41bb-8c5c-1e78fe85beb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9"
Oct 30 19:00:18.924761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.924734    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" event={"ID":"ddb6c9b7-0581-42e2-9632-144ee021879b","Type":"ContainerStarted","Data":"24b5b6a684c2c1387a4e3b3d9a8e84931836384cf367884c8c924eda289e84dd"}
Oct 30 19:00:18.938764 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.938735    4267 scope.go:122] "RemoveContainer" containerID="34daa1b796c79387aada0157db0b281ef6313b8a35fd95f412f10ebc0ee9f037"
Oct 30 19:00:18.954373 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:18.954347    4267 scope.go:122] "RemoveContainer" containerID="ad4c2183023bb8b77134ff2b08a50e33b4ff5167c2ecb49308fba654e961ac53"
Oct 30 19:00:19.026895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:19.026853    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"]
Oct 30 19:00:19.138465 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:19.138433    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"]
Oct 30 19:00:19.254965 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:19.254928    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"]
Oct 30 19:00:19.936340 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:19.936311    4267 generic.go:356] "Generic (PLEG): container finished" podID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerID="24b5b6a684c2c1387a4e3b3d9a8e84931836384cf367884c8c924eda289e84dd" exitCode=60
Oct 30 19:00:19.936814 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:19.936381    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" event={"ID":"ddb6c9b7-0581-42e2-9632-144ee021879b","Type":"ContainerDied","Data":"24b5b6a684c2c1387a4e3b3d9a8e84931836384cf367884c8c924eda289e84dd"}
Oct 30 19:00:19.936814 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:19.936440    4267 scope.go:122] "RemoveContainer" containerID="3c5d93ac1c8396041f4b4de081fd2fd978a2771e8d7dea121980ef1d6dd2f964"
Oct 30 19:00:19.936926 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:19.936869    4267 scope.go:122] "RemoveContainer" containerID="24b5b6a684c2c1387a4e3b3d9a8e84931836384cf367884c8c924eda289e84dd"
Oct 30 19:00:19.937250 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:19.937224    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72_odh-ai-gateway-infra(ddb6c9b7-0581-42e2-9632-144ee021879b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b"
Oct 30 19:00:19.993536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:19.993157    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"]
Oct 30 19:00:19.998695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:19.998668    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.057744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.057256    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"]
Oct 30 19:00:20.084534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.082774    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.084534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.082829    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.084534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.082849    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.084534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.082869    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.084534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.082939    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.084534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.082971    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.084534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.082987    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.084534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.083053    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.084534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.083086    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bp6\" (UniqueName: \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-kube-api-access-c9bp6\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.162346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.162256    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"
Oct 30 19:00:20.178289 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:20.178226    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:00:20.184432 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.184399    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.184586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.184462    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bp6\" (UniqueName: \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-kube-api-access-c9bp6\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.184672 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.184596    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.184672 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.184643    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.184788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.184669    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.184788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.184701    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.184909 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.184800    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.184909 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.184845    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.184909 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.184871    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.185857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.185138    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-certs\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.185857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.185220    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-credential-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.186541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.186273    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-socket\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.186541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.186444    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-data\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.187039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.187013    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istiod-ca-cert\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.192067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.192042    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-podinfo\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.200324 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.200291    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-envoy\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.209181 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.209155    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"
Oct 30 19:00:20.217615 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.217586    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"
Oct 30 19:00:20.242341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.242308    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-token\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.242606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.242582    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bp6\" (UniqueName: \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-kube-api-access-c9bp6\") pod \"e2e-worker-w5-611053-openshift-default-5cf969d899-74b65\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") " pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.267733 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.267692    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"]
Oct 30 19:00:20.269697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.269665    4267 cpu_manager.go:414] "containerMap: removing container" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerName="cleanup"
Oct 30 19:00:20.269868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.269854    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerName="cleanup"
Oct 30 19:00:20.269960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.269950    4267 cpu_manager.go:414] "containerMap: removing container" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerName="cleanup"
Oct 30 19:00:20.270056 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.270046    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerName="cleanup"
Oct 30 19:00:20.270137 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.270128    4267 cpu_manager.go:414] "containerMap: removing container" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerName="cleanup"
Oct 30 19:00:20.270205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.270196    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerName="cleanup"
Oct 30 19:00:20.270286 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.270276    4267 cpu_manager.go:414] "containerMap: removing container" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerName="cleanup"
Oct 30 19:00:20.270354 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.270345    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.284608    4267 cpu_manager.go:414] "containerMap: removing container" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.284646    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.284911    4267 cpu_manager.go:414] "containerMap: removing container" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.284922    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.285099    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.285298    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.285311    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.285320    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.285334    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.285350    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerName="cleanup"
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.285782    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-maas-api-service-ca\") pod \"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9\" (UID: \"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9\") "
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.285833    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b5b0a1-ea85-49d5-96a1-b417620ffacb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b5b0a1-ea85-49d5-96a1-b417620ffacb-maas-api-service-ca\") pod \"19b5b0a1-ea85-49d5-96a1-b417620ffacb\" (UID: \"19b5b0a1-ea85-49d5-96a1-b417620ffacb\") "
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.285866    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-kube-api-access-qlkbn\" (UniqueName: \"kubernetes.io/projected/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-kube-api-access-qlkbn\") pod \"dd2f7f58-73ee-4675-9d18-b6d8595e2af5\" (UID: \"dd2f7f58-73ee-4675-9d18-b6d8595e2af5\") "
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.286003    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-kube-api-access-9svr9\" (UniqueName: \"kubernetes.io/projected/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-kube-api-access-9svr9\") pod \"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9\" (UID: \"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9\") "
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.286141    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b5b0a1-ea85-49d5-96a1-b417620ffacb-kube-api-access-wsp7r\" (UniqueName: \"kubernetes.io/projected/19b5b0a1-ea85-49d5-96a1-b417620ffacb-kube-api-access-wsp7r\") pod \"19b5b0a1-ea85-49d5-96a1-b417620ffacb\" (UID: \"19b5b0a1-ea85-49d5-96a1-b417620ffacb\") "
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.286179    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-maas-api-service-ca\") pod \"dd2f7f58-73ee-4675-9d18-b6d8595e2af5\" (UID: \"dd2f7f58-73ee-4675-9d18-b6d8595e2af5\") "
Oct 30 19:00:20.286538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.286245    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-maas-api-service-ca" pod "34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" (UID: "34fa5b52-0e16-41bb-8c5c-1e78fe85beb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:20.287513 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.286831    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.287513 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.287256    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b5b0a1-ea85-49d5-96a1-b417620ffacb-maas-api-service-ca" pod "19b5b0a1-ea85-49d5-96a1-b417620ffacb" (UID: "19b5b0a1-ea85-49d5-96a1-b417620ffacb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:20.287513 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.287476    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-maas-api-service-ca" pod "dd2f7f58-73ee-4675-9d18-b6d8595e2af5" (UID: "dd2f7f58-73ee-4675-9d18-b6d8595e2af5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:20.289888 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.289862    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.290466 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.290340    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-kube-api-access-9svr9" pod "34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" (UID: "34fa5b52-0e16-41bb-8c5c-1e78fe85beb9"). InnerVolumeSpecName "kube-api-access-9svr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:20.291033 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.291005    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-kube-api-access-qlkbn" pod "dd2f7f58-73ee-4675-9d18-b6d8595e2af5" (UID: "dd2f7f58-73ee-4675-9d18-b6d8595e2af5"). InnerVolumeSpecName "kube-api-access-qlkbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:20.291244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.291231    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b5b0a1-ea85-49d5-96a1-b417620ffacb-kube-api-access-wsp7r" pod "19b5b0a1-ea85-49d5-96a1-b417620ffacb" (UID: "19b5b0a1-ea85-49d5-96a1-b417620ffacb"). InnerVolumeSpecName "kube-api-access-wsp7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:20.316899 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.316869    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:20.339746 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.339260    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"]
Oct 30 19:00:20.388339 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.388295    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.388552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.388466    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.388636 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.388557    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.388636 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.388586    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfcx\" (UniqueName: \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-kube-api-access-7dfcx\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.388731 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.388643    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.388781 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.388738    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec59e1d-c104-4fca-9793-1a55b8d16680-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.388839 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.388785    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.388896 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.388838    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.388896 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.388879    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.389021 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.388998    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b5b0a1-ea85-49d5-96a1-b417620ffacb-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.389085 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.389021    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlkbn\" (UniqueName: \"kubernetes.io/projected/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-kube-api-access-qlkbn\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.389085 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.389037    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9svr9\" (UniqueName: \"kubernetes.io/projected/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9-kube-api-access-9svr9\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.389085 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.389052    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsp7r\" (UniqueName: \"kubernetes.io/projected/19b5b0a1-ea85-49d5-96a1-b417620ffacb-kube-api-access-wsp7r\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.389085 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.389066    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f7f58-73ee-4675-9d18-b6d8595e2af5-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.392436 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.391745    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"]
Oct 30 19:00:20.490085 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.489993    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490085 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490053    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490098    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490117    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfcx\" (UniqueName: \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-kube-api-access-7dfcx\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490153    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490215    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec59e1d-c104-4fca-9793-1a55b8d16680-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490246    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490429    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490438    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-certs\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490525    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490569    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-credential-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490525    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-data\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.490809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.490775    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-socket\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.491072 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.491050    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec59e1d-c104-4fca-9793-1a55b8d16680-istiod-ca-cert\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.492592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.492571    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-envoy\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.492806 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.492790    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-podinfo\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.546172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.546130    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-token\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.546328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.546276    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfcx\" (UniqueName: \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-kube-api-access-7dfcx\") pod \"e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") " pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.604941 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.604905    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:20.768420 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.768337    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"]
Oct 30 19:00:20.781836 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:00:20.781787    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c7ce5b_c4f5_4a31_807f_e2ba352d5302.slice/crio-475b22597918a65c9d3c5fc51120396fd04e8fa0b0461450bdca7d6656e43f30 WatchSource:0}: Error finding container 475b22597918a65c9d3c5fc51120396fd04e8fa0b0461450bdca7d6656e43f30: Status 404 returned error can't find the container with id 475b22597918a65c9d3c5fc51120396fd04e8fa0b0461450bdca7d6656e43f30
Oct 30 19:00:20.784865 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.784823    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:20.784985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.784901    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:20.784985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.784939    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:20.965687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.959760    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5" event={"ID":"19b5b0a1-ea85-49d5-96a1-b417620ffacb","Type":"ContainerDied","Data":"b5876e38b7face50f615a7d588f78df121319cdadd2ec5195c8659fc9988f446"}
Oct 30 19:00:20.965687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.959813    4267 scope.go:122] "RemoveContainer" containerID="e3e857d768fed2aef7fdc922c02e51dc79cdcae4c4d44621f5ed06588c56ff4b"
Oct 30 19:00:20.965687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.959942    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"
Oct 30 19:00:20.973961 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.973930    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw" event={"ID":"dd2f7f58-73ee-4675-9d18-b6d8595e2af5","Type":"ContainerDied","Data":"b7e082ecb39311dc156f9293523cf8b288d37281fb74816491dc69287d728612"}
Oct 30 19:00:20.974084 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.974053    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"
Oct 30 19:00:20.976769 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.976662    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" event={"ID":"09c7ce5b-c4f5-4a31-807f-e2ba352d5302","Type":"ContainerStarted","Data":"e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a"}
Oct 30 19:00:20.976769 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.976721    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" event={"ID":"09c7ce5b-c4f5-4a31-807f-e2ba352d5302","Type":"ContainerStarted","Data":"475b22597918a65c9d3c5fc51120396fd04e8fa0b0461450bdca7d6656e43f30"}
Oct 30 19:00:20.979845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.979808    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls" event={"ID":"34fa5b52-0e16-41bb-8c5c-1e78fe85beb9","Type":"ContainerDied","Data":"66727a1e8c1ef6177ac41759d61d4701c78a2cc70db27cf1e9fbaaa1aa02c270"}
Oct 30 19:00:20.979945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.979925    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"
Oct 30 19:00:21.006713 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:20.993213    4267 scope.go:122] "RemoveContainer" containerID="e97e42bffc596f1c7d054158ec6c2ebf90a00c1ab359e63201a31537c101a40e"
Oct 30 19:00:21.006713 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.004721    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:21.006713 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.004814    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:21.006713 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.004861    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:21.013968 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.013939    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"]
Oct 30 19:00:21.022223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.022192    4267 scope.go:122] "RemoveContainer" containerID="b35c3a94a8c677bdecd1905710a7ecb4ddc65e0183dc5dc17a51a908e8cd795e"
Oct 30 19:00:21.191262 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.189566    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"]
Oct 30 19:00:21.194314 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.194291    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5763a-29889780-j46ls"]
Oct 30 19:00:21.222259 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.222230    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"
Oct 30 19:00:21.232530 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.232471    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" path="/var/lib/kubelet/pods/34fa5b52-0e16-41bb-8c5c-1e78fe85beb9/volumes"
Oct 30 19:00:21.304801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.304718    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb6c9b7-0581-42e2-9632-144ee021879b-kube-api-access-g7plc\" (UniqueName: \"kubernetes.io/projected/ddb6c9b7-0581-42e2-9632-144ee021879b-kube-api-access-g7plc\") pod \"ddb6c9b7-0581-42e2-9632-144ee021879b\" (UID: \"ddb6c9b7-0581-42e2-9632-144ee021879b\") "
Oct 30 19:00:21.304982 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.304857    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb6c9b7-0581-42e2-9632-144ee021879b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb6c9b7-0581-42e2-9632-144ee021879b-maas-api-service-ca\") pod \"ddb6c9b7-0581-42e2-9632-144ee021879b\" (UID: \"ddb6c9b7-0581-42e2-9632-144ee021879b\") "
Oct 30 19:00:21.305258 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.305234    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb6c9b7-0581-42e2-9632-144ee021879b-maas-api-service-ca" pod "ddb6c9b7-0581-42e2-9632-144ee021879b" (UID: "ddb6c9b7-0581-42e2-9632-144ee021879b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:21.305637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.305614    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb6c9b7-0581-42e2-9632-144ee021879b-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.306594 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.306543    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podStartSLOduration=2.3065286 podStartE2EDuration="2.3065286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:21.301807053 +0000 UTC m=+1312.189724139" watchObservedRunningTime="2026-10-30 19:00:21.3065286 +0000 UTC m=+1312.194445690"
Oct 30 19:00:21.306905 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.306883    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb6c9b7-0581-42e2-9632-144ee021879b-kube-api-access-g7plc" pod "ddb6c9b7-0581-42e2-9632-144ee021879b" (UID: "ddb6c9b7-0581-42e2-9632-144ee021879b"). InnerVolumeSpecName "kube-api-access-g7plc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:21.318051 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.318021    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:21.319765 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.319740    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:00:21.319887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.319795    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:00:21.407277 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.407245    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7plc\" (UniqueName: \"kubernetes.io/projected/ddb6c9b7-0581-42e2-9632-144ee021879b-kube-api-access-g7plc\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.440543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.440457    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"]
Oct 30 19:00:21.446749 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.446716    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc98e1-29889780-jpdjw"]
Oct 30 19:00:21.662360 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.662266    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"]
Oct 30 19:00:21.665955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:21.665926    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1290a6-29889780-jbsm5"]
Oct 30 19:00:22.005549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.005430    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" event={"ID":"dec59e1d-c104-4fca-9793-1a55b8d16680","Type":"ContainerStarted","Data":"70308c602f44297543e956a5a6a4f74e787c8a7215b2d502ef56bac88d608f27"}
Oct 30 19:00:22.005996 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.005581    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" event={"ID":"dec59e1d-c104-4fca-9793-1a55b8d16680","Type":"ContainerStarted","Data":"04142a8fdab4badf0350fef1c0031e95fac5d8f0099cc9a2a2718442e960d52a"}
Oct 30 19:00:22.006949 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.006928    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"
Oct 30 19:00:22.006949 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.006934    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72" event={"ID":"ddb6c9b7-0581-42e2-9632-144ee021879b","Type":"ContainerDied","Data":"446086258a59083293d615f2f73cde4210534236f913dadc747b4e360ad6fb94"}
Oct 30 19:00:22.007168 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.006980    4267 scope.go:122] "RemoveContainer" containerID="24b5b6a684c2c1387a4e3b3d9a8e84931836384cf367884c8c924eda289e84dd"
Oct 30 19:00:22.170936 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.170574    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" podStartSLOduration=2.170552095 podStartE2EDuration="2.170552095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:22.128818104 +0000 UTC m=+1313.016735194" watchObservedRunningTime="2026-10-30 19:00:22.170552095 +0000 UTC m=+1313.058469184"
Oct 30 19:00:22.270064 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.269947    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"]
Oct 30 19:00:22.274527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.274482    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-41db7c-29889780-wzw72"]
Oct 30 19:00:22.337518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.322295    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:00:22.337518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.322357    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:00:22.606036 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.605965    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:22.606991 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.606968    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:00:22.607155 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:22.607137    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 19:00:23.272389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:23.272351    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" path="/var/lib/kubelet/pods/19b5b0a1-ea85-49d5-96a1-b417620ffacb/volumes"
Oct 30 19:00:23.272925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:23.272915    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" path="/var/lib/kubelet/pods/dd2f7f58-73ee-4675-9d18-b6d8595e2af5/volumes"
Oct 30 19:00:23.273332 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:23.273315    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b" path="/var/lib/kubelet/pods/ddb6c9b7-0581-42e2-9632-144ee021879b/volumes"
Oct 30 19:00:23.327517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:23.322932    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:00:23.327517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:23.322998    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:00:23.605786 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:23.605698    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:00:23.605965 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:23.605778    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 19:00:24.323517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:24.318830    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:00:24.323517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:24.318982    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:00:24.605418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:24.605342    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:00:24.605651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:24.605629    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 19:00:25.105604 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:25.105568    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:25.184963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:25.184934    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:25.327517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:25.318071    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:00:25.327517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:25.318142    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:00:25.606003 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:25.605972    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:00:25.606249 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:25.606227    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 19:00:26.102924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:26.102894    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:26.320959 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:26.320919    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:00:26.337835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:26.337786    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:00:26.605475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:26.605440    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:00:26.605670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:26.605525    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 19:00:27.318112 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:27.318065    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:00:27.318335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:27.318148    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:00:27.606210 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:27.606110    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 19:00:27.606210 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:27.606197    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 19:00:28.653928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:28.653895    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:28.655536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:28.655508    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:28.656278 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:28.656254    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:00:28.960152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:28.960104    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:28.961416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:28.961377    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:28.963115 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:28.963098    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:00:34.529519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.494265    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.663327 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.663267    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:00:34.689282 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.689234    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.711518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.711455    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.908657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.908558    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"]
Oct 30 19:00:34.911132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.911101    4267 cpu_manager.go:414] "containerMap: removing container" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerName="cleanup"
Oct 30 19:00:34.911979 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.911951    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerName="cleanup"
Oct 30 19:00:34.912169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.912154    4267 cpu_manager.go:414] "containerMap: removing container" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerName="cleanup"
Oct 30 19:00:34.912258 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.912246    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerName="cleanup"
Oct 30 19:00:34.912352 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.912341    4267 cpu_manager.go:414] "containerMap: removing container" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerName="cleanup"
Oct 30 19:00:34.912438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.912427    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerName="cleanup"
Oct 30 19:00:34.912586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.912573    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerName="cleanup"
Oct 30 19:00:34.912690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.912678    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerName="cleanup"
Oct 30 19:00:34.912782 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.912772    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerName="cleanup"
Oct 30 19:00:34.912858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.912848    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerName="cleanup"
Oct 30 19:00:34.912944 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.912933    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerName="cleanup"
Oct 30 19:00:34.913017 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.913007    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerName="cleanup"
Oct 30 19:00:34.913276 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.913260    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b5b0a1-ea85-49d5-96a1-b417620ffacb" containerName="cleanup"
Oct 30 19:00:34.920578 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.920548    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerName="cleanup"
Oct 30 19:00:34.920750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.920736    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="34fa5b52-0e16-41bb-8c5c-1e78fe85beb9" containerName="cleanup"
Oct 30 19:00:34.920835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.920825    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerName="cleanup"
Oct 30 19:00:34.920945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.920934    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb6c9b7-0581-42e2-9632-144ee021879b" containerName="cleanup"
Oct 30 19:00:34.921050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.921039    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2f7f58-73ee-4675-9d18-b6d8595e2af5" containerName="cleanup"
Oct 30 19:00:34.933809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.933775    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:34.963516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.956874    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.976064 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.975232    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"]
Oct 30 19:00:34.991739 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.991695    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.994635 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.994599    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:34.999565 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:34.995971    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.027649 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.003173    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"]
Oct 30 19:00:35.102134 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.099250    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.148518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.147945    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.165574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.164166    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.164278    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.189877    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc708cda-e0e7-4c57-99e9-81c0d158077a-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.189937    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.189970    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.190031    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.190056    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.190099    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.190141    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.190260    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.190292    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.190334    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5c2d3f-4778-470b-8cc3-3f96f7529302-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.190406    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcrwz\" (UniqueName: \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-kube-api-access-qcrwz\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.195518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.190469    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.257518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.252627    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.298517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.296639    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czskk\" (UniqueName: \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-kube-api-access-czskk\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.298517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.296703    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.298517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.296744    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.298517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.296826    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.364518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.342421    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"]
Oct 30 19:00:35.401637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.401582    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcrwz\" (UniqueName: \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-kube-api-access-qcrwz\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.402004 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.401969    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.402244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.402211    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czskk\" (UniqueName: \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-kube-api-access-czskk\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.403955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.403930    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.404759 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.404484    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.405722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.404952    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.405958 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.405916    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.406541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.406521    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.406900 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.406841    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.406900 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.406867    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.407458    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc708cda-e0e7-4c57-99e9-81c0d158077a-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.407625    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.407713    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.407847    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-data\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.407988    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.408044    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.408255    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.408469    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.408620    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.408908    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-credential-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.409065    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.410516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.409102    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.413582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.413556    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5c2d3f-4778-470b-8cc3-3f96f7529302-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.416179 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.415763    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.416179 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.415043    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-certs\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.417190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.414418    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5c2d3f-4778-470b-8cc3-3f96f7529302-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.417359 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.417183    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc708cda-e0e7-4c57-99e9-81c0d158077a-istiod-ca-cert\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.417546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.417307    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-socket\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.423612 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.423024    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.424268 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.423928    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.424268 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.424210    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-podinfo\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.488519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.484735    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-envoy\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.530094 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.498247    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czskk\" (UniqueName: \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-kube-api-access-czskk\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.530094 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.499448    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcrwz\" (UniqueName: \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-kube-api-access-qcrwz\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.530094 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.525702    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.530094 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.530031    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-token\") pod \"e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") " pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.558526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.548584    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.558526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.549381    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:00:35.718650 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.718602    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:00:35.778518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.720795    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:35.821808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.821596    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.822555 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.783984    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.824171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.824144    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:00:35.846515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.785246    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 19:00:35.846820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.807773    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:35.851604 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.782777    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:00:35.864561 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.852164    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:00:35.958520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.784016    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:00:35.958520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.957775    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:00:35.994291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.994249    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.994962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.994937    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:00:35.996055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.995580    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:35.996543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:35.996525    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:00:36.017660 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.017615    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:36.018278 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.018255    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:00:36.143226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.135575    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:00:36.153618 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.153572    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:36.154220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.154197    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:00:36.285134 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.285046    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:00:36.367788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.360047    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:36.368427 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.368402    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:00:36.621856 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.621812    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:36.714913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.714440    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:00:36.748486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.748415    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:00:36.766906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.764584    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:00:36.776130 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.776099    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:00:36.784423 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.784394    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:00:36.827539 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.826596    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:00:36.828776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:36.828081    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:00:37.050164 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.050126    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"]
Oct 30 19:00:37.054841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.054814    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b6b7dad-nvglb"]
Oct 30 19:00:37.161322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.161295    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"]
Oct 30 19:00:37.165501 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.165467    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"]
Oct 30 19:00:37.168367 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:00:37.168335    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5c2d3f_4778_470b_8cc3_3f96f7529302.slice/crio-8edeb36182dd59e2a482b121f86f79dad5cbff8dfc7993343d829a5283f1ad7d WatchSource:0}: Error finding container 8edeb36182dd59e2a482b121f86f79dad5cbff8dfc7993343d829a5283f1ad7d: Status 404 returned error can't find the container with id 8edeb36182dd59e2a482b121f86f79dad5cbff8dfc7993343d829a5283f1ad7d
Oct 30 19:00:37.170194 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:00:37.170166    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc708cda_e0e7_4c57_99e9_81c0d158077a.slice/crio-b95cac5aee91c47138555182e3850e6f14780f1283dfa2b5c2ba11e86028fd57 WatchSource:0}: Error finding container b95cac5aee91c47138555182e3850e6f14780f1283dfa2b5c2ba11e86028fd57: Status 404 returned error can't find the container with id b95cac5aee91c47138555182e3850e6f14780f1283dfa2b5c2ba11e86028fd57
Oct 30 19:00:37.171534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.171454    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:37.171615 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.171573    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:37.171690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.171617    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:37.172681 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.172641    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:37.172800 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.172732    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:37.172800 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.172772    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:37.226534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.226447    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7638e1-be67-4e71-ab34-32b05fb4c8dd" path="/var/lib/kubelet/pods/3d7638e1-be67-4e71-ab34-32b05fb4c8dd/volumes"
Oct 30 19:00:37.844099 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.844042    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" event={"ID":"df5c2d3f-4778-470b-8cc3-3f96f7529302","Type":"ContainerStarted","Data":"943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2"}
Oct 30 19:00:37.844744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.844698    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" event={"ID":"df5c2d3f-4778-470b-8cc3-3f96f7529302","Type":"ContainerStarted","Data":"8edeb36182dd59e2a482b121f86f79dad5cbff8dfc7993343d829a5283f1ad7d"}
Oct 30 19:00:37.853155 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.852384    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" event={"ID":"fc708cda-e0e7-4c57-99e9-81c0d158077a","Type":"ContainerStarted","Data":"9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77"}
Oct 30 19:00:37.853155 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:37.852442    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" event={"ID":"fc708cda-e0e7-4c57-99e9-81c0d158077a","Type":"ContainerStarted","Data":"b95cac5aee91c47138555182e3850e6f14780f1283dfa2b5c2ba11e86028fd57"}
Oct 30 19:00:38.302158 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:38.302106    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" podStartSLOduration=4.30209188 podStartE2EDuration="4.30209188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:38.299709598 +0000 UTC m=+1329.187626714" watchObservedRunningTime="2026-10-30 19:00:38.30209188 +0000 UTC m=+1329.190008968"
Oct 30 19:00:38.450676 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:38.450608    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" podStartSLOduration=4.450584994 podStartE2EDuration="4.450584994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:38.436833183 +0000 UTC m=+1329.324750276" watchObservedRunningTime="2026-10-30 19:00:38.450584994 +0000 UTC m=+1329.338502085"
Oct 30 19:00:38.783450 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:38.783421    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:38.784693 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:38.784664    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:38.784798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:38.784726    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:00:38.828826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:38.828796    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:38.838557 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:38.834614    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:38.838557 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:38.834673    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:00:39.783525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:39.782068    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:39.783525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:39.782141    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:00:39.831517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:39.830063    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:39.831517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:39.830123    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:00:40.782515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:40.782141    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:40.782515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:40.782213    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:00:40.826512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:40.824730    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:40.826512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:40.824894    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:00:41.782607 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:41.782567    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:41.782796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:41.782617    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:00:41.825518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:41.822142    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:41.825518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:41.822220    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:00:42.782826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:42.782769    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:42.783353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:42.782829    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:00:42.839535 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:42.808472    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:42.839897 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:42.839854    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:00:43.782950 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:43.782911    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:00:43.783609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:43.783565    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:00:43.809097 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:43.809065    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:00:43.809293 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:43.809134    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:00:44.910479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:44.910443    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:44.911778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:44.911753    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:44.914038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:44.914019    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:00:45.013987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:45.013952    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:45.014389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:45.014361    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:45.016478 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:45.016457    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:00:49.979940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:49.979905    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"]
Oct 30 19:00:49.983927 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:49.983908    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.093296 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.093247    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.093525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.093335    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.093525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.093360    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.093525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.093404    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt2lh\" (UniqueName: \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-kube-api-access-wt2lh\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.093525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.093435    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.093778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.093558    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.093778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.093608    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.093778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.093635    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebcad52-7747-4d1c-be69-61193c14e5e3-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.093778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.093751    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.095977 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.095948    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"]
Oct 30 19:00:50.162035 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:00:50.162004    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:00:50.194823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.194787    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195017 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.194840    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195017 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.194866    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebcad52-7747-4d1c-be69-61193c14e5e3-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195017 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.194965    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195192 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.195066    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195192 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.195128    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195192 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.195152    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.195198    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt2lh\" (UniqueName: \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-kube-api-access-wt2lh\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.195233    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.195315    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195758 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.195724    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195758 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.195739    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.195921 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.195767    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebcad52-7747-4d1c-be69-61193c14e5e3-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.196233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.196211    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.197353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.197326    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.198040 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.198017    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.253349 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.253267    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.253550 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.253425    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt2lh\" (UniqueName: \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-kube-api-access-wt2lh\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.298109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.298074    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:50.743880 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.743843    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"]
Oct 30 19:00:50.745943 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:00:50.745914    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebcad52_7747_4d1c_be69_61193c14e5e3.slice/crio-0752db37a653b36d38f2326381cb060b1b54c916c9a4aef0d83f8c41e190af03 WatchSource:0}: Error finding container 0752db37a653b36d38f2326381cb060b1b54c916c9a4aef0d83f8c41e190af03: Status 404 returned error can't find the container with id 0752db37a653b36d38f2326381cb060b1b54c916c9a4aef0d83f8c41e190af03
Oct 30 19:00:50.749303 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.749257    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:50.749426 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.749375    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:50.749483 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:50.749442    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:00:51.019727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.019465    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" event={"ID":"0ebcad52-7747-4d1c-be69-61193c14e5e3","Type":"ContainerStarted","Data":"021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd"}
Oct 30 19:00:51.019727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.019555    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" event={"ID":"0ebcad52-7747-4d1c-be69-61193c14e5e3","Type":"ContainerStarted","Data":"0752db37a653b36d38f2326381cb060b1b54c916c9a4aef0d83f8c41e190af03"}
Oct 30 19:00:51.158872 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.158730    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podStartSLOduration=2.15870628 podStartE2EDuration="2.15870628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:51.140944269 +0000 UTC m=+1342.028861361" watchObservedRunningTime="2026-10-30 19:00:51.15870628 +0000 UTC m=+1342.046623369"
Oct 30 19:00:51.299865 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.299327    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:51.301196 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.301160    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:51.301321 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.301222    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:00:51.723516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.722907    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"]
Oct 30 19:00:51.735521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.734790    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:51.795515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.788247    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:51.821519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.810545    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:00:51.821519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.811716    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"]
Oct 30 19:00:51.845549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.841878    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttn4n\" (UniqueName: \"kubernetes.io/projected/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-kube-api-access-ttn4n\") pod \"maas-api-revoke-sub-5aba1b03-2mlhs\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:51.845549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.841973    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aba1b03-2mlhs\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:51.845549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.842130    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aba1b03-2mlhs\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:52.018671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.999091    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aba1b03-2mlhs\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:52.018671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.999295    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttn4n\" (UniqueName: \"kubernetes.io/projected/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-kube-api-access-ttn4n\") pod \"maas-api-revoke-sub-5aba1b03-2mlhs\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:52.018671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:51.999367    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aba1b03-2mlhs\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:52.018671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:52.005223    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aba1b03-2mlhs\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:52.175519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:52.153021    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aba1b03-2mlhs\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:52.328585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:52.307140    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:52.328585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:52.307200    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:00:52.328585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:52.308235    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttn4n\" (UniqueName: \"kubernetes.io/projected/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-kube-api-access-ttn4n\") pod \"maas-api-revoke-sub-5aba1b03-2mlhs\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:52.395516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:52.385758    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:53.406579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.381623    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:53.406579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.381689    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:00:53.527811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.503271    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:00:53.684303 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.627648    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:53.785261 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.785216    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:53.798763 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.798720    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:00:53.837549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.836337    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:53.855548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.850213    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:53.859010 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.858972    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:00:53.861849 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.859313    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:00:53.900399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:53.899669    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:54.148573 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.131009    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"]
Oct 30 19:00:54.149592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.149556    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:54.176667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.176623    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:54.345519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.312656    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:54.635642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.635586    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:00:54.843386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.636204    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.863955    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.947922    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.056193    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.056263    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.321193    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.968528    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.980417    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.010783    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.012763    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.052108    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:54.980659    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:00:55.095962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.003906    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:55.175572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.175265    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:00:55.198474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.188700    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:55.198474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.189129    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:00:55.198474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.189280    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:00:55.207528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.200597    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:00:55.207528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.200688    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:00:55.207528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.200753    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:00:55.207528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.200821    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:00:55.207528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.201095    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:55.207528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.201172    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:00:55.431165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.378260    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:55.470679 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.470631    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:00:55.596554 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.594768    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:00:55.596554 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.596098    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs" event={"ID":"75ad2ed9-30cc-4e03-9bd9-5d358127aef5","Type":"ContainerStarted","Data":"0c8a34c492e35365286749002d08a3d78a793639662dfe3d7512715380c4904a"}
Oct 30 19:00:55.602783 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.602668    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:00:55.602783 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.602721    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:00:55.604776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.604754    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:00:55.605198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.604807    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:00:55.605198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.604840    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:00:55.605198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.604872    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:00:55.605198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.604902    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:00:55.606127 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.605346    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:00:55.656255 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.656210    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:00:55.659952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:55.659921    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:00:56.299681 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:56.299646    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:56.299849 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:56.299702    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:00:56.658213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:56.658124    4267 generic.go:356] "Generic (PLEG): container finished" podID="75ad2ed9-30cc-4e03-9bd9-5d358127aef5" containerID="9d17d3db0762850a2e7c71c9fa0b95f3930801568cd7c0911ded0642d873fc6d" exitCode=0
Oct 30 19:00:56.659724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:56.659697    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs" event={"ID":"75ad2ed9-30cc-4e03-9bd9-5d358127aef5","Type":"ContainerDied","Data":"9d17d3db0762850a2e7c71c9fa0b95f3930801568cd7c0911ded0642d873fc6d"}
Oct 30 19:00:57.299488 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:57.299457    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:57.299703 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:57.299555    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:00:57.906580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:57.906541    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:57.983338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:57.983297    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-xks-ca\") pod \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") "
Oct 30 19:00:57.983586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:57.983453    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-openshift-ca\") pod \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") "
Oct 30 19:00:57.983666 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:57.983580    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-kube-api-access-ttn4n\" (UniqueName: \"kubernetes.io/projected/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-kube-api-access-ttn4n\") pod \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\" (UID: \"75ad2ed9-30cc-4e03-9bd9-5d358127aef5\") "
Oct 30 19:00:57.984955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:57.984915    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-openshift-ca" pod "75ad2ed9-30cc-4e03-9bd9-5d358127aef5" (UID: "75ad2ed9-30cc-4e03-9bd9-5d358127aef5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:57.987697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:57.987660    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-xks-ca" pod "75ad2ed9-30cc-4e03-9bd9-5d358127aef5" (UID: "75ad2ed9-30cc-4e03-9bd9-5d358127aef5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:57.987802 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:57.987689    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-kube-api-access-ttn4n" pod "75ad2ed9-30cc-4e03-9bd9-5d358127aef5" (UID: "75ad2ed9-30cc-4e03-9bd9-5d358127aef5"). InnerVolumeSpecName "kube-api-access-ttn4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:58.086358 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:58.086327    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:58.086358 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:58.086354    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:58.086599 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:58.086363    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttn4n\" (UniqueName: \"kubernetes.io/projected/75ad2ed9-30cc-4e03-9bd9-5d358127aef5-kube-api-access-ttn4n\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:00:58.299077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:58.298991    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:00:58.299077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:58.299040    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:00:58.691818 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:58.691788    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"
Oct 30 19:00:58.691978 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:58.691809    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs" event={"ID":"75ad2ed9-30cc-4e03-9bd9-5d358127aef5","Type":"ContainerDied","Data":"0c8a34c492e35365286749002d08a3d78a793639662dfe3d7512715380c4904a"}
Oct 30 19:00:58.691978 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:58.691904    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c8a34c492e35365286749002d08a3d78a793639662dfe3d7512715380c4904a"
Oct 30 19:00:59.623695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:59.621168    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:59.623695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:59.622718    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:00:59.625244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:00:59.625219    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:01:00.598379 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.598349    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"]
Oct 30 19:01:00.598963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.598946    4267 cpu_manager.go:414] "containerMap: removing container" podUID="75ad2ed9-30cc-4e03-9bd9-5d358127aef5" containerName="revoke-keys"
Oct 30 19:01:00.599022 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.598964    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ad2ed9-30cc-4e03-9bd9-5d358127aef5" containerName="revoke-keys"
Oct 30 19:01:00.599090 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.599079    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ad2ed9-30cc-4e03-9bd9-5d358127aef5" containerName="revoke-keys"
Oct 30 19:01:00.602786 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.602765    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:00.656960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.656922    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:00.657846 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.657691    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:01:00.675757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.675722    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"]
Oct 30 19:01:00.725185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.725156    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a3abf53-86vkw\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:00.725374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.725284    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a3abf53-86vkw\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:00.725374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.725313    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvld\" (UniqueName: \"kubernetes.io/projected/e3f3bd02-6ba1-420a-add2-e2068d1796ca-kube-api-access-grvld\") pod \"maas-api-revoke-sub-7a3abf53-86vkw\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:00.827331 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.827296    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a3abf53-86vkw\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:00.827548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.827467    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a3abf53-86vkw\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:00.827620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.827546    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvld\" (UniqueName: \"kubernetes.io/projected/e3f3bd02-6ba1-420a-add2-e2068d1796ca-kube-api-access-grvld\") pod \"maas-api-revoke-sub-7a3abf53-86vkw\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:00.828140 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.828114    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a3abf53-86vkw\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:00.829801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.829781    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a3abf53-86vkw\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:00.884381 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.884304    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvld\" (UniqueName: \"kubernetes.io/projected/e3f3bd02-6ba1-420a-add2-e2068d1796ca-kube-api-access-grvld\") pod \"maas-api-revoke-sub-7a3abf53-86vkw\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:00.923173 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:00.923135    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:01.306601 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:01:01.306566    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f3bd02_6ba1_420a_add2_e2068d1796ca.slice/crio-ef69f96a2a72da75b0ce55d2db198307ef48d0313ceab3c5004ca26adf644bbc WatchSource:0}: Error finding container ef69f96a2a72da75b0ce55d2db198307ef48d0313ceab3c5004ca26adf644bbc: Status 404 returned error can't find the container with id ef69f96a2a72da75b0ce55d2db198307ef48d0313ceab3c5004ca26adf644bbc
Oct 30 19:01:01.308736 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:01.308710    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"]
Oct 30 19:01:01.720970 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:01.720931    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw" event={"ID":"e3f3bd02-6ba1-420a-add2-e2068d1796ca","Type":"ContainerStarted","Data":"ef69f96a2a72da75b0ce55d2db198307ef48d0313ceab3c5004ca26adf644bbc"}
Oct 30 19:01:02.730223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:02.730190    4267 generic.go:356] "Generic (PLEG): container finished" podID="e3f3bd02-6ba1-420a-add2-e2068d1796ca" containerID="3803572ee900abf0b45f00c571b6f2ff37e600e49ed573ef98ef94697e4e220b" exitCode=0
Oct 30 19:01:02.730652 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:02.730279    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw" event={"ID":"e3f3bd02-6ba1-420a-add2-e2068d1796ca","Type":"ContainerDied","Data":"3803572ee900abf0b45f00c571b6f2ff37e600e49ed573ef98ef94697e4e220b"}
Oct 30 19:01:03.911689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:03.911656    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:04.075395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.075314    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f3bd02-6ba1-420a-add2-e2068d1796ca-kube-api-access-grvld\" (UniqueName: \"kubernetes.io/projected/e3f3bd02-6ba1-420a-add2-e2068d1796ca-kube-api-access-grvld\") pod \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") "
Oct 30 19:01:04.075395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.075393    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-openshift-ca\") pod \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") "
Oct 30 19:01:04.075625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.075582    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-xks-ca\") pod \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\" (UID: \"e3f3bd02-6ba1-420a-add2-e2068d1796ca\") "
Oct 30 19:01:04.075854 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.075829    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-openshift-ca" pod "e3f3bd02-6ba1-420a-add2-e2068d1796ca" (UID: "e3f3bd02-6ba1-420a-add2-e2068d1796ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:04.076180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.076159    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:04.077722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.077697    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f3bd02-6ba1-420a-add2-e2068d1796ca-kube-api-access-grvld" pod "e3f3bd02-6ba1-420a-add2-e2068d1796ca" (UID: "e3f3bd02-6ba1-420a-add2-e2068d1796ca"). InnerVolumeSpecName "kube-api-access-grvld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:04.077722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.077712    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-xks-ca" pod "e3f3bd02-6ba1-420a-add2-e2068d1796ca" (UID: "e3f3bd02-6ba1-420a-add2-e2068d1796ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:04.177705 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.177663    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grvld\" (UniqueName: \"kubernetes.io/projected/e3f3bd02-6ba1-420a-add2-e2068d1796ca-kube-api-access-grvld\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:04.177888 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.177707    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f3bd02-6ba1-420a-add2-e2068d1796ca-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:04.747987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.747955    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"
Oct 30 19:01:04.748213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.747961    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw" event={"ID":"e3f3bd02-6ba1-420a-add2-e2068d1796ca","Type":"ContainerDied","Data":"ef69f96a2a72da75b0ce55d2db198307ef48d0313ceab3c5004ca26adf644bbc"}
Oct 30 19:01:04.748213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.748089    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef69f96a2a72da75b0ce55d2db198307ef48d0313ceab3c5004ca26adf644bbc"
Oct 30 19:01:04.943154 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.943120    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"]
Oct 30 19:01:04.944027 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.944004    4267 cpu_manager.go:414] "containerMap: removing container" podUID="e3f3bd02-6ba1-420a-add2-e2068d1796ca" containerName="revoke-keys"
Oct 30 19:01:04.944075 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.944031    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f3bd02-6ba1-420a-add2-e2068d1796ca" containerName="revoke-keys"
Oct 30 19:01:04.944191 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.944179    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f3bd02-6ba1-420a-add2-e2068d1796ca" containerName="revoke-keys"
Oct 30 19:01:04.948366 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:04.948345    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.000868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.000787    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:05.000868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.000856    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:01:05.003654 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.003623    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"]
Oct 30 19:01:05.089840 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.089806    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp9ng\" (UniqueName: \"kubernetes.io/projected/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-kube-api-access-hp9ng\") pod \"maas-api-revoke-sub-c7ae76b8-kvxsd\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.089986 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.089864    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ae76b8-kvxsd\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.090043 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.090019    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ae76b8-kvxsd\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.095754 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.095727    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"]
Oct 30 19:01:05.099796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.099776    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.160381 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.160341    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"]
Oct 30 19:01:05.191666 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.191626    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.191826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.191673    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.191826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.191771    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.192003 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.191835    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pctw\" (UniqueName: \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-kube-api-access-2pctw\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.192003 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.191896    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.192003 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.191949    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.192003 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.191997    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.192213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.192056    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.192213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.192101    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp9ng\" (UniqueName: \"kubernetes.io/projected/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-kube-api-access-hp9ng\") pod \"maas-api-revoke-sub-c7ae76b8-kvxsd\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.192213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.192165    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ae76b8-kvxsd\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.192369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.192221    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.192369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.192290    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ae76b8-kvxsd\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.193160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.193136    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ae76b8-kvxsd\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.195979 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.195955    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ae76b8-kvxsd\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.258200 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.258127    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp9ng\" (UniqueName: \"kubernetes.io/projected/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-kube-api-access-hp9ng\") pod \"maas-api-revoke-sub-c7ae76b8-kvxsd\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.264316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.264295    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:05.294358 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.294326    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.294536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.294407    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.294613 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.294548    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.294690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.294672    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.294747 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.294708    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.294808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.294752    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.294808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.294759    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.294808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.294787    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pctw\" (UniqueName: \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-kube-api-access-2pctw\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.294973 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.294849    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.294973 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.294917    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.295092 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.295071    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.295187 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.295166    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.295251 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.295176    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.295308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.295272    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.297404 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.297376    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.298031 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.298009    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.357449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.357410    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pctw\" (UniqueName: \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-kube-api-access-2pctw\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.360579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.360553    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.419848 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.419561    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:05.653765 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:01:05.653736    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcbaa21_bc20_4fc9_ba01_68d2dcff5c85.slice/crio-85f48a860d92fce9b115d5c6d2e90be7a5c37f8f8aabc9f5168824ea6194144a WatchSource:0}: Error finding container 85f48a860d92fce9b115d5c6d2e90be7a5c37f8f8aabc9f5168824ea6194144a: Status 404 returned error can't find the container with id 85f48a860d92fce9b115d5c6d2e90be7a5c37f8f8aabc9f5168824ea6194144a
Oct 30 19:01:05.656232 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.656207    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"]
Oct 30 19:01:05.756574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.756533    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd" event={"ID":"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85","Type":"ContainerStarted","Data":"85f48a860d92fce9b115d5c6d2e90be7a5c37f8f8aabc9f5168824ea6194144a"}
Oct 30 19:01:05.805260 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:01:05.805230    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bc8497_e8fc_491b_bf9b_7cf563607f3c.slice/crio-31f49bf01683f29f02c3ce07997cec49c28d38c0d0d68a07d004c077319a7b4f WatchSource:0}: Error finding container 31f49bf01683f29f02c3ce07997cec49c28d38c0d0d68a07d004c077319a7b4f: Status 404 returned error can't find the container with id 31f49bf01683f29f02c3ce07997cec49c28d38c0d0d68a07d004c077319a7b4f
Oct 30 19:01:05.805839 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.805756    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"]
Oct 30 19:01:05.807890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.807849    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:01:05.807975 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.807932    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:01:05.808009 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:05.807981    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:01:06.765434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:06.765340    4267 generic.go:356] "Generic (PLEG): container finished" podID="3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85" containerID="494e391086fc96cd2ef09cf1ee710ed62c9c47cc855744ddc20f0bc55d6beee5" exitCode=0
Oct 30 19:01:06.765434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:06.765387    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd" event={"ID":"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85","Type":"ContainerDied","Data":"494e391086fc96cd2ef09cf1ee710ed62c9c47cc855744ddc20f0bc55d6beee5"}
Oct 30 19:01:06.767082 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:06.767059    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" event={"ID":"c1bc8497-e8fc-491b-bf9b-7cf563607f3c","Type":"ContainerStarted","Data":"694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463"}
Oct 30 19:01:06.767191 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:06.767106    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" event={"ID":"c1bc8497-e8fc-491b-bf9b-7cf563607f3c","Type":"ContainerStarted","Data":"31f49bf01683f29f02c3ce07997cec49c28d38c0d0d68a07d004c077319a7b4f"}
Oct 30 19:01:06.999601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:06.999545    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" podStartSLOduration=1.999530239 podStartE2EDuration="1.999530239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:06.996101304 +0000 UTC m=+1357.884018392" watchObservedRunningTime="2026-10-30 19:01:06.999530239 +0000 UTC m=+1357.887447327"
Oct 30 19:01:07.431517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:07.427869    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:07.431517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:07.430028    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:01:07.431517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:07.430081    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:01:07.991870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:07.991847    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:08.157689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.157528    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-xks-ca\") pod \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") "
Oct 30 19:01:08.158056 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.158029    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-openshift-ca\") pod \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") "
Oct 30 19:01:08.158333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.158314    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-kube-api-access-hp9ng\" (UniqueName: \"kubernetes.io/projected/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-kube-api-access-hp9ng\") pod \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\" (UID: \"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85\") "
Oct 30 19:01:08.158604 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.158575    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-openshift-ca" pod "3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85" (UID: "3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:08.159369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.159345    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:08.161786 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.161747    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-xks-ca" pod "3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85" (UID: "3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:08.164292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.164256    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-kube-api-access-hp9ng" pod "3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85" (UID: "3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85"). InnerVolumeSpecName "kube-api-access-hp9ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:08.260725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.260691    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:08.260725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.260724    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp9ng\" (UniqueName: \"kubernetes.io/projected/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85-kube-api-access-hp9ng\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:08.432577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.431927    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:01:08.432577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.432007    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:01:08.786502 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.786457    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"
Oct 30 19:01:08.786708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.786508    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd" event={"ID":"3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85","Type":"ContainerDied","Data":"85f48a860d92fce9b115d5c6d2e90be7a5c37f8f8aabc9f5168824ea6194144a"}
Oct 30 19:01:08.786708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:08.786590    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85f48a860d92fce9b115d5c6d2e90be7a5c37f8f8aabc9f5168824ea6194144a"
Oct 30 19:01:09.426517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:09.423889    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:01:09.426517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:09.423954    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:01:10.425695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:10.425645    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:01:10.426015 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:10.425983    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:01:11.426517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:11.422827    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:01:11.426517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:11.422898    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:01:12.420738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.420697    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 19:01:12.420898 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.420764    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 19:01:12.515404 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.515366    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"]
Oct 30 19:01:12.516262 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.516240    4267 cpu_manager.go:414] "containerMap: removing container" podUID="3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85" containerName="revoke-keys"
Oct 30 19:01:12.516306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.516268    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85" containerName="revoke-keys"
Oct 30 19:01:12.516485 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.516470    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85" containerName="revoke-keys"
Oct 30 19:01:12.520816 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.520795    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:12.580707 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.580656    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"]
Oct 30 19:01:12.584587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.581583    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:12.621706 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.621655    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:01:12.640761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.640720    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5412588-bb2x4\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:12.641315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.641289    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hqh\" (UniqueName: \"kubernetes.io/projected/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-kube-api-access-s6hqh\") pod \"maas-api-revoke-sub-b5412588-bb2x4\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:12.641465 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.641447    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5412588-bb2x4\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:12.743877 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.743826    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hqh\" (UniqueName: \"kubernetes.io/projected/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-kube-api-access-s6hqh\") pod \"maas-api-revoke-sub-b5412588-bb2x4\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:12.744060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.743887    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5412588-bb2x4\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:12.744060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.743942    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5412588-bb2x4\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:12.744635 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.744609    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5412588-bb2x4\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:12.746342 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.746322    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5412588-bb2x4\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:12.800246 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.800215    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hqh\" (UniqueName: \"kubernetes.io/projected/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-kube-api-access-s6hqh\") pod \"maas-api-revoke-sub-b5412588-bb2x4\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:12.837970 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:12.837937    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:13.332611 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:13.332570    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"]
Oct 30 19:01:13.563665 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:01:13.563621    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18c7bdf_6ce0_4e0a_af52_a0db4ea7e749.slice/crio-702cc4f80e038d51203ee04ac76eb97ddfe99455b2c71592520728599bacc58e WatchSource:0}: Error finding container 702cc4f80e038d51203ee04ac76eb97ddfe99455b2c71592520728599bacc58e: Status 404 returned error can't find the container with id 702cc4f80e038d51203ee04ac76eb97ddfe99455b2c71592520728599bacc58e
Oct 30 19:01:13.602527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:13.601376    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1e703f6-pqhvd"]
Oct 30 19:01:13.617522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:13.606143    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"]
Oct 30 19:01:14.201746 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.201693    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4" event={"ID":"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749","Type":"ContainerStarted","Data":"702cc4f80e038d51203ee04ac76eb97ddfe99455b2c71592520728599bacc58e"}
Oct 30 19:01:14.311768 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.306610    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.501638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.501599    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:14.501910 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.501879    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.603242 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.603185    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.604387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.604337    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.604989 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.604862    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.605382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.605359    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.613140 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.613089    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.613770 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.613733    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.614205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.614182    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:01:14.616442 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.603124    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:01:14.657175 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.657118    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.661634 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.661576    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:14.701649 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:14.701600    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.285205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.285149    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bba0c3b-340e-45d5-ba1e-9b7c2666f309" path="/var/lib/kubelet/pods/6bba0c3b-340e-45d5-ba1e-9b7c2666f309/volumes"
Oct 30 19:01:15.316532 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.314640    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.316532 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.315088    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:01:15.388161 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.388077    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4" event={"ID":"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749","Type":"ContainerStarted","Data":"08d626924e30055dd6ccd93e7d57a4509f9126c896f40a4bff9e7cdfe01954c8"}
Oct 30 19:01:15.729608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.591179    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4" podStartSLOduration=3.591155723 podStartE2EDuration="3.591155723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:15.576953823 +0000 UTC m=+1366.464870914" watchObservedRunningTime="2026-10-30 19:01:15.591155723 +0000 UTC m=+1366.479072812"
Oct 30 19:01:15.729608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.624769    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.729608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.728621    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.624807    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.624830    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.624867    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.774190    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.774463    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.774626    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.624907    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.624887    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.676836    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.677536    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.776570    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.705111    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.685656    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.624959    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:01:15.784519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.676732    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.852520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.676812    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:15.852520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:15.826605    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:16.007837 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.007731    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:01:16.060571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.032306    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:01:16.060571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.032801    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:01:16.060571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.032905    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:01:16.060571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.033093    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:01:16.060571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.034360    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:01:16.060571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.034447    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:01:16.158907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.118850    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:16.158907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.119748    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:16.158907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.119832    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:16.158907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.155320    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:01:16.334298 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.325448    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:16.885295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:16.885241    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:01:17.081467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.076918    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:17.081467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.078151    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:01:17.081467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.079185    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:01:17.081467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.080111    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:01:17.283337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.283284    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:17.396755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.396686    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:17.436521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.411853    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:01:17.795515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.445166    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:17.840316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.525935    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:01:17.873526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.841002    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": context deadline exceeded"
Oct 30 19:01:18.031517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.526339    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:01:18.031517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.526144    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:01:18.031517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:17.888665    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.036755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.036699    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.180574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.163660    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.180574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.164094    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:01:18.180574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.164362    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:01:18.180574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.164630    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:01:18.253518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.198756    4267 generic.go:356] "Generic (PLEG): container finished" podID="a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749" containerID="08d626924e30055dd6ccd93e7d57a4509f9126c896f40a4bff9e7cdfe01954c8" exitCode=0
Oct 30 19:01:18.253518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.200133    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:18.253518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.200185    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.426582 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:01:18.395957    4267 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.112s"
Oct 30 19:01:18.426582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.422651    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4" event={"ID":"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749","Type":"ContainerDied","Data":"08d626924e30055dd6ccd93e7d57a4509f9126c896f40a4bff9e7cdfe01954c8"}
Oct 30 19:01:18.539523 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.538483    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:01:18.894672 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.894568    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.925545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.895205    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.925545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.895296    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.949615 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:18.895989    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.081518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.018976    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.136887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.057642    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.198708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.192429    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.199579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.199534    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.230423 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.200128    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.230423 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.200364    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.230423 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.200654    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.300651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.300600    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.571769 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.571662    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.959924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.959867    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.976907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.960633    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.977449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.960684    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.977873 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.961019    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.994934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:19.994887    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Oct 30 19:01:20.171516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.163867    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:20.171516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.167817    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:20.217522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.217389    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:20.227516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.217800    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:20.229170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.223729    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:01:20.229660 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.223779    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:01:20.230027 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.223802    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:01:20.248061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.248011    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:20.352530 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:01:20.352441    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:01:20.382307 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.381425    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:01:20.493604 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.490885    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:01:20.496559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.495863    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:01:20.697271 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.694104    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:01:20.810081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.804744    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:01:20.810081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.804829    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:01:20.810081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.804884    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:01:20.810081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.804932    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:01:20.819694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.816808    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:01:20.819694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.818763    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:01:20.819694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.819678    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:01:20.828091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.821989    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:01:20.869351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.869322    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:01:20.939093 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:20.939061    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:21.030173 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.030138    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-openshift-ca\") pod \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") "
Oct 30 19:01:21.030360 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.030223    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-xks-ca\") pod \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") "
Oct 30 19:01:21.030433 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.030368    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-kube-api-access-s6hqh\" (UniqueName: \"kubernetes.io/projected/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-kube-api-access-s6hqh\") pod \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\" (UID: \"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749\") "
Oct 30 19:01:21.030606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.030577    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-openshift-ca" pod "a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749" (UID: "a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:21.031103 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.031077    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:21.032638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.032612    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-kube-api-access-s6hqh" pod "a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749" (UID: "a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749"). InnerVolumeSpecName "kube-api-access-s6hqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:21.032800 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.032780    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-xks-ca" pod "a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749" (UID: "a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:21.132862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.132775    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:21.132862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.132810    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6hqh\" (UniqueName: \"kubernetes.io/projected/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749-kube-api-access-s6hqh\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:21.806406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.806376    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4" event={"ID":"a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749","Type":"ContainerDied","Data":"702cc4f80e038d51203ee04ac76eb97ddfe99455b2c71592520728599bacc58e"}
Oct 30 19:01:21.806812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.806423    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="702cc4f80e038d51203ee04ac76eb97ddfe99455b2c71592520728599bacc58e"
Oct 30 19:01:21.806812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:21.806427    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"
Oct 30 19:01:23.228477 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.228443    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"]
Oct 30 19:01:23.229225 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.229202    4267 cpu_manager.go:414] "containerMap: removing container" podUID="a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749" containerName="revoke-keys"
Oct 30 19:01:23.229333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.229226    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749" containerName="revoke-keys"
Oct 30 19:01:23.229439 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.229426    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749" containerName="revoke-keys"
Oct 30 19:01:23.233213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.233182    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.277729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.277700    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be6cbb63-kk4k2\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.278007 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.277981    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be6cbb63-kk4k2\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.278170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.278145    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9qj\" (UniqueName: \"kubernetes.io/projected/84b1f8d9-ab15-4999-bd24-9990568778fd-kube-api-access-mb9qj\") pod \"maas-api-revoke-sub-be6cbb63-kk4k2\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.289429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.289405    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:23.289671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.289628    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:01:23.290637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.290615    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"]
Oct 30 19:01:23.312966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.312935    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"]
Oct 30 19:01:23.317916 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.317893    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.379265 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.379234    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"]
Oct 30 19:01:23.379825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.379804    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be6cbb63-kk4k2\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.379888 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.379848    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48f7e613-bpw7m\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.379953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.379943    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9qj\" (UniqueName: \"kubernetes.io/projected/84b1f8d9-ab15-4999-bd24-9990568778fd-kube-api-access-mb9qj\") pod \"maas-api-revoke-sub-be6cbb63-kk4k2\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.380070 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.380047    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d589r\" (UniqueName: \"kubernetes.io/projected/4c5a987b-e25c-4832-9921-7ed50b5321d4-kube-api-access-d589r\") pod \"maas-api-revoke-sub-48f7e613-bpw7m\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.380140 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.380121    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48f7e613-bpw7m\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.380217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.380204    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be6cbb63-kk4k2\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.381033 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.381008    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be6cbb63-kk4k2\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.384873 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.384852    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be6cbb63-kk4k2\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.447833 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.447793    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9qj\" (UniqueName: \"kubernetes.io/projected/84b1f8d9-ab15-4999-bd24-9990568778fd-kube-api-access-mb9qj\") pod \"maas-api-revoke-sub-be6cbb63-kk4k2\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.482272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.482188    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48f7e613-bpw7m\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.482418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.482357    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d589r\" (UniqueName: \"kubernetes.io/projected/4c5a987b-e25c-4832-9921-7ed50b5321d4-kube-api-access-d589r\") pod \"maas-api-revoke-sub-48f7e613-bpw7m\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.482418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.482411    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48f7e613-bpw7m\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.483056 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.483033    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48f7e613-bpw7m\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.484676 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.484657    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48f7e613-bpw7m\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.539502 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.539453    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d589r\" (UniqueName: \"kubernetes.io/projected/4c5a987b-e25c-4832-9921-7ed50b5321d4-kube-api-access-d589r\") pod \"maas-api-revoke-sub-48f7e613-bpw7m\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.565324 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.565300    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:23.636978 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.635998    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:23.943709 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:01:23.943681    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b1f8d9_ab15_4999_bd24_9990568778fd.slice/crio-d826fbec68e729ebee2413dcbc539fdfed7de774d9b2e0bcc2656e8d553c828b WatchSource:0}: Error finding container d826fbec68e729ebee2413dcbc539fdfed7de774d9b2e0bcc2656e8d553c828b: Status 404 returned error can't find the container with id d826fbec68e729ebee2413dcbc539fdfed7de774d9b2e0bcc2656e8d553c828b
Oct 30 19:01:23.946316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:23.946289    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"]
Oct 30 19:01:24.041692 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:01:24.041667    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5a987b_e25c_4832_9921_7ed50b5321d4.slice/crio-02ff6f328ce9cb3a4104c70522e3c28839d500691134b54cf79cdce64743ec10 WatchSource:0}: Error finding container 02ff6f328ce9cb3a4104c70522e3c28839d500691134b54cf79cdce64743ec10: Status 404 returned error can't find the container with id 02ff6f328ce9cb3a4104c70522e3c28839d500691134b54cf79cdce64743ec10
Oct 30 19:01:24.044991 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:24.044966    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"]
Oct 30 19:01:24.832470 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:24.832435    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2" event={"ID":"84b1f8d9-ab15-4999-bd24-9990568778fd","Type":"ContainerStarted","Data":"e446950d7e87c9c1462c1635a465f215541641ec5dbaa9bd604d56ef2d86c88b"}
Oct 30 19:01:24.832917 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:24.832508    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2" event={"ID":"84b1f8d9-ab15-4999-bd24-9990568778fd","Type":"ContainerStarted","Data":"d826fbec68e729ebee2413dcbc539fdfed7de774d9b2e0bcc2656e8d553c828b"}
Oct 30 19:01:24.834305 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:24.834279    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m" event={"ID":"4c5a987b-e25c-4832-9921-7ed50b5321d4","Type":"ContainerStarted","Data":"46ad484d67b1464e1560164284537233f08c984b73d4d688091acb260df1e8cf"}
Oct 30 19:01:24.834429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:24.834318    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m" event={"ID":"4c5a987b-e25c-4832-9921-7ed50b5321d4","Type":"ContainerStarted","Data":"02ff6f328ce9cb3a4104c70522e3c28839d500691134b54cf79cdce64743ec10"}
Oct 30 19:01:24.948046 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:24.947263    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2" podStartSLOduration=1.9472437820000001 podStartE2EDuration="1.947243782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:24.946201477 +0000 UTC m=+1375.834118567" watchObservedRunningTime="2026-10-30 19:01:24.947243782 +0000 UTC m=+1375.835160872"
Oct 30 19:01:25.061963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:25.061910    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m" podStartSLOduration=2.06189536 podStartE2EDuration="2.06189536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:25.0589427 +0000 UTC m=+1375.946859789" watchObservedRunningTime="2026-10-30 19:01:25.06189536 +0000 UTC m=+1375.949812498"
Oct 30 19:01:25.844509 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:25.844462    4267 generic.go:356] "Generic (PLEG): container finished" podID="84b1f8d9-ab15-4999-bd24-9990568778fd" containerID="e446950d7e87c9c1462c1635a465f215541641ec5dbaa9bd604d56ef2d86c88b" exitCode=0
Oct 30 19:01:25.844962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:25.844528    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2" event={"ID":"84b1f8d9-ab15-4999-bd24-9990568778fd","Type":"ContainerDied","Data":"e446950d7e87c9c1462c1635a465f215541641ec5dbaa9bd604d56ef2d86c88b"}
Oct 30 19:01:25.846593 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:25.846565    4267 generic.go:356] "Generic (PLEG): container finished" podID="4c5a987b-e25c-4832-9921-7ed50b5321d4" containerID="46ad484d67b1464e1560164284537233f08c984b73d4d688091acb260df1e8cf" exitCode=0
Oct 30 19:01:25.846718 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:25.846600    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m" event={"ID":"4c5a987b-e25c-4832-9921-7ed50b5321d4","Type":"ContainerDied","Data":"46ad484d67b1464e1560164284537233f08c984b73d4d688091acb260df1e8cf"}
Oct 30 19:01:27.038435 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.038412    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:27.073631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.073609    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:27.141029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.140954    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5a987b-e25c-4832-9921-7ed50b5321d4-kube-api-access-d589r\" (UniqueName: \"kubernetes.io/projected/4c5a987b-e25c-4832-9921-7ed50b5321d4-kube-api-access-d589r\") pod \"4c5a987b-e25c-4832-9921-7ed50b5321d4\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") "
Oct 30 19:01:27.141029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.141026    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-xks-ca\") pod \"4c5a987b-e25c-4832-9921-7ed50b5321d4\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") "
Oct 30 19:01:27.141273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.141078    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-xks-ca\") pod \"84b1f8d9-ab15-4999-bd24-9990568778fd\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") "
Oct 30 19:01:27.141273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.141100    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-openshift-ca\") pod \"84b1f8d9-ab15-4999-bd24-9990568778fd\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") "
Oct 30 19:01:27.141273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.141196    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b1f8d9-ab15-4999-bd24-9990568778fd-kube-api-access-mb9qj\" (UniqueName: \"kubernetes.io/projected/84b1f8d9-ab15-4999-bd24-9990568778fd-kube-api-access-mb9qj\") pod \"84b1f8d9-ab15-4999-bd24-9990568778fd\" (UID: \"84b1f8d9-ab15-4999-bd24-9990568778fd\") "
Oct 30 19:01:27.141273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.141255    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-openshift-ca\") pod \"4c5a987b-e25c-4832-9921-7ed50b5321d4\" (UID: \"4c5a987b-e25c-4832-9921-7ed50b5321d4\") "
Oct 30 19:01:27.141600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.141573    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-openshift-ca" pod "84b1f8d9-ab15-4999-bd24-9990568778fd" (UID: "84b1f8d9-ab15-4999-bd24-9990568778fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:27.141779 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.141663    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-openshift-ca" pod "4c5a987b-e25c-4832-9921-7ed50b5321d4" (UID: "4c5a987b-e25c-4832-9921-7ed50b5321d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:27.142120 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.142096    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:27.142209 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.142126    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:27.143273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.143250    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-xks-ca" pod "84b1f8d9-ab15-4999-bd24-9990568778fd" (UID: "84b1f8d9-ab15-4999-bd24-9990568778fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:27.143719 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.143697    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5a987b-e25c-4832-9921-7ed50b5321d4-kube-api-access-d589r" pod "4c5a987b-e25c-4832-9921-7ed50b5321d4" (UID: "4c5a987b-e25c-4832-9921-7ed50b5321d4"). InnerVolumeSpecName "kube-api-access-d589r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:27.143799 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.143728    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b1f8d9-ab15-4999-bd24-9990568778fd-kube-api-access-mb9qj" pod "84b1f8d9-ab15-4999-bd24-9990568778fd" (UID: "84b1f8d9-ab15-4999-bd24-9990568778fd"). InnerVolumeSpecName "kube-api-access-mb9qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:27.143799 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.143768    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-xks-ca" pod "4c5a987b-e25c-4832-9921-7ed50b5321d4" (UID: "4c5a987b-e25c-4832-9921-7ed50b5321d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:27.243051 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.243022    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b1f8d9-ab15-4999-bd24-9990568778fd-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:27.243051 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.243049    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb9qj\" (UniqueName: \"kubernetes.io/projected/84b1f8d9-ab15-4999-bd24-9990568778fd-kube-api-access-mb9qj\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:27.243190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.243060    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d589r\" (UniqueName: \"kubernetes.io/projected/4c5a987b-e25c-4832-9921-7ed50b5321d4-kube-api-access-d589r\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:27.243190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.243069    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5a987b-e25c-4832-9921-7ed50b5321d4-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:27.864333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.864299    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"
Oct 30 19:01:27.864543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.864335    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m" event={"ID":"4c5a987b-e25c-4832-9921-7ed50b5321d4","Type":"ContainerDied","Data":"02ff6f328ce9cb3a4104c70522e3c28839d500691134b54cf79cdce64743ec10"}
Oct 30 19:01:27.864543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.864383    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02ff6f328ce9cb3a4104c70522e3c28839d500691134b54cf79cdce64743ec10"
Oct 30 19:01:27.865892 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.865875    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"
Oct 30 19:01:27.866009 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.865913    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2" event={"ID":"84b1f8d9-ab15-4999-bd24-9990568778fd","Type":"ContainerDied","Data":"d826fbec68e729ebee2413dcbc539fdfed7de774d9b2e0bcc2656e8d553c828b"}
Oct 30 19:01:27.866009 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:27.865949    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d826fbec68e729ebee2413dcbc539fdfed7de774d9b2e0bcc2656e8d553c828b"
Oct 30 19:01:31.815631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.815589    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"]
Oct 30 19:01:31.816165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.816152    4267 cpu_manager.go:414] "containerMap: removing container" podUID="4c5a987b-e25c-4832-9921-7ed50b5321d4" containerName="revoke-keys"
Oct 30 19:01:31.816211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.816166    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5a987b-e25c-4832-9921-7ed50b5321d4" containerName="revoke-keys"
Oct 30 19:01:31.816211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.816189    4267 cpu_manager.go:414] "containerMap: removing container" podUID="84b1f8d9-ab15-4999-bd24-9990568778fd" containerName="revoke-keys"
Oct 30 19:01:31.816211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.816195    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b1f8d9-ab15-4999-bd24-9990568778fd" containerName="revoke-keys"
Oct 30 19:01:31.816304 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.816294    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5a987b-e25c-4832-9921-7ed50b5321d4" containerName="revoke-keys"
Oct 30 19:01:31.816341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.816304    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b1f8d9-ab15-4999-bd24-9990568778fd" containerName="revoke-keys"
Oct 30 19:01:31.820861 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.820842    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:31.902827 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.902784    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"]
Oct 30 19:01:31.907943 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.907911    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghqb\" (UniqueName: \"kubernetes.io/projected/f2b9780b-61d8-4ce2-a151-219c29e5d967-kube-api-access-nghqb\") pod \"maas-api-revoke-sub-704d3027-qtmnb\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:31.908086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.907990    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704d3027-qtmnb\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:31.908131 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.908114    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704d3027-qtmnb\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:31.922552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.922521    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:31.922712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:31.922692    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:01:32.009778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:32.009739    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghqb\" (UniqueName: \"kubernetes.io/projected/f2b9780b-61d8-4ce2-a151-219c29e5d967-kube-api-access-nghqb\") pod \"maas-api-revoke-sub-704d3027-qtmnb\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:32.009976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:32.009835    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704d3027-qtmnb\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:32.009976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:32.009904    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704d3027-qtmnb\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:32.010526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:32.010468    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704d3027-qtmnb\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:32.012280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:32.012244    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704d3027-qtmnb\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:32.067911 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:32.067835    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghqb\" (UniqueName: \"kubernetes.io/projected/f2b9780b-61d8-4ce2-a151-219c29e5d967-kube-api-access-nghqb\") pod \"maas-api-revoke-sub-704d3027-qtmnb\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:32.137970 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:32.137933    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:32.522838 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:01:32.522803    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b9780b_61d8_4ce2_a151_219c29e5d967.slice/crio-aabb8a835e35ba092d6cee5a19534b5366e211182446a0813c490f5762a3622a WatchSource:0}: Error finding container aabb8a835e35ba092d6cee5a19534b5366e211182446a0813c490f5762a3622a: Status 404 returned error can't find the container with id aabb8a835e35ba092d6cee5a19534b5366e211182446a0813c490f5762a3622a
Oct 30 19:01:32.523180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:32.523155    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"]
Oct 30 19:01:32.915565 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:32.915528    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb" event={"ID":"f2b9780b-61d8-4ce2-a151-219c29e5d967","Type":"ContainerStarted","Data":"d87563cab01a898ac05ad2d94d02133598b79c24a197d9cc6395e65d3f9051d3"}
Oct 30 19:01:32.915983 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:32.915624    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb" event={"ID":"f2b9780b-61d8-4ce2-a151-219c29e5d967","Type":"ContainerStarted","Data":"aabb8a835e35ba092d6cee5a19534b5366e211182446a0813c490f5762a3622a"}
Oct 30 19:01:33.032693 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:33.032641    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb" podStartSLOduration=2.032625711 podStartE2EDuration="2.032625711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:33.030174933 +0000 UTC m=+1383.918092024" watchObservedRunningTime="2026-10-30 19:01:33.032625711 +0000 UTC m=+1383.920542799"
Oct 30 19:01:33.925604 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:33.925565    4267 generic.go:356] "Generic (PLEG): container finished" podID="f2b9780b-61d8-4ce2-a151-219c29e5d967" containerID="d87563cab01a898ac05ad2d94d02133598b79c24a197d9cc6395e65d3f9051d3" exitCode=0
Oct 30 19:01:33.926092 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:33.925664    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb" event={"ID":"f2b9780b-61d8-4ce2-a151-219c29e5d967","Type":"ContainerDied","Data":"d87563cab01a898ac05ad2d94d02133598b79c24a197d9cc6395e65d3f9051d3"}
Oct 30 19:01:34.677677 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.677634    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"]
Oct 30 19:01:34.681821 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.681796    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.740812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.740777    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:01:34.741043 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.740969    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:01:34.756419 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.756382    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.756614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.756506    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.756673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.756629    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxf4\" (UniqueName: \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-kube-api-access-7cxf4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.756712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.756680    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.756743 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.756708    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.756869 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.756848    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.756915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.756881    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.790428 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.790381    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:01:34.790716 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.790509    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5sqr9\""
Oct 30 19:01:34.801400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.801361    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"]
Oct 30 19:01:34.858007 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.857974    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.858221 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.858089    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxf4\" (UniqueName: \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-kube-api-access-7cxf4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.858221 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.858169    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.858221 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.858206    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.858394 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.858240    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.858394 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.858265    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.858394 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.858352    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.859129 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.859103    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.861073 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.861039    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.861161 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.861042    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.861239 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.861224    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.864703 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.864676    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.915466 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.915430    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.915810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.915783    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxf4\" (UniqueName: \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-kube-api-access-7cxf4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jdgbq\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:34.995580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:34.995539    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:35.139968 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.139936    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:35.170545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.164083    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b9780b-61d8-4ce2-a151-219c29e5d967-kube-api-access-nghqb\" (UniqueName: \"kubernetes.io/projected/f2b9780b-61d8-4ce2-a151-219c29e5d967-kube-api-access-nghqb\") pod \"f2b9780b-61d8-4ce2-a151-219c29e5d967\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") "
Oct 30 19:01:35.170545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.164147    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-xks-ca\") pod \"f2b9780b-61d8-4ce2-a151-219c29e5d967\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") "
Oct 30 19:01:35.170545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.164219    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-openshift-ca\") pod \"f2b9780b-61d8-4ce2-a151-219c29e5d967\" (UID: \"f2b9780b-61d8-4ce2-a151-219c29e5d967\") "
Oct 30 19:01:35.170545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.165262    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-openshift-ca" pod "f2b9780b-61d8-4ce2-a151-219c29e5d967" (UID: "f2b9780b-61d8-4ce2-a151-219c29e5d967"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:35.170545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.169231    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b9780b-61d8-4ce2-a151-219c29e5d967-kube-api-access-nghqb" pod "f2b9780b-61d8-4ce2-a151-219c29e5d967" (UID: "f2b9780b-61d8-4ce2-a151-219c29e5d967"). InnerVolumeSpecName "kube-api-access-nghqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:35.170545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.170453    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-xks-ca" pod "f2b9780b-61d8-4ce2-a151-219c29e5d967" (UID: "f2b9780b-61d8-4ce2-a151-219c29e5d967"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:35.266804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.266774    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nghqb\" (UniqueName: \"kubernetes.io/projected/f2b9780b-61d8-4ce2-a151-219c29e5d967-kube-api-access-nghqb\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:35.266804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.266806    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:35.266990 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.266816    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b9780b-61d8-4ce2-a151-219c29e5d967-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:35.542935 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:01:35.542905    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ea6e52_1ff7_4ca4_9c9c_6081c0f1a908.slice/crio-6ca54e208957d58175226ffbb194f7504189c02c77152f617201fa639fdc90de WatchSource:0}: Error finding container 6ca54e208957d58175226ffbb194f7504189c02c77152f617201fa639fdc90de: Status 404 returned error can't find the container with id 6ca54e208957d58175226ffbb194f7504189c02c77152f617201fa639fdc90de
Oct 30 19:01:35.544220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.544191    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"]
Oct 30 19:01:35.545792 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.545773    4267 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:01:35.944554 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.944522    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"
Oct 30 19:01:35.944754 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.944548    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb" event={"ID":"f2b9780b-61d8-4ce2-a151-219c29e5d967","Type":"ContainerDied","Data":"aabb8a835e35ba092d6cee5a19534b5366e211182446a0813c490f5762a3622a"}
Oct 30 19:01:35.944754 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.944606    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aabb8a835e35ba092d6cee5a19534b5366e211182446a0813c490f5762a3622a"
Oct 30 19:01:35.946218 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:35.946181    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq" event={"ID":"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908","Type":"ContainerStarted","Data":"6ca54e208957d58175226ffbb194f7504189c02c77152f617201fa639fdc90de"}
Oct 30 19:01:38.635173 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.635135    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"]
Oct 30 19:01:38.635830 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.635812    4267 cpu_manager.go:414] "containerMap: removing container" podUID="f2b9780b-61d8-4ce2-a151-219c29e5d967" containerName="revoke-keys"
Oct 30 19:01:38.635889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.635831    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b9780b-61d8-4ce2-a151-219c29e5d967" containerName="revoke-keys"
Oct 30 19:01:38.635943 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.635934    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b9780b-61d8-4ce2-a151-219c29e5d967" containerName="revoke-keys"
Oct 30 19:01:38.639558 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.639536    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:38.698590 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.698558    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"]
Oct 30 19:01:38.722622 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.722578    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a46943a-sb8f9\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:38.722804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.722626    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a46943a-sb8f9\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:38.722882 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.722805    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cls9r\" (UniqueName: \"kubernetes.io/projected/cee017bb-6870-45aa-baf0-933740e18bf5-kube-api-access-cls9r\") pod \"maas-api-revoke-sub-7a46943a-sb8f9\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:38.741627 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.741535    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:38.741627 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.741582    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:01:38.824597 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.824560    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cls9r\" (UniqueName: \"kubernetes.io/projected/cee017bb-6870-45aa-baf0-933740e18bf5-kube-api-access-cls9r\") pod \"maas-api-revoke-sub-7a46943a-sb8f9\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:38.824821 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.824798    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a46943a-sb8f9\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:38.824925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.824829    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a46943a-sb8f9\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:38.825687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.825664    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a46943a-sb8f9\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:38.827409 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.827387    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a46943a-sb8f9\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:38.882433 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.882398    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cls9r\" (UniqueName: \"kubernetes.io/projected/cee017bb-6870-45aa-baf0-933740e18bf5-kube-api-access-cls9r\") pod \"maas-api-revoke-sub-7a46943a-sb8f9\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:38.957005 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:38.956761    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:39.359699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:39.359670    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"]
Oct 30 19:01:39.360930 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:01:39.360899    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee017bb_6870_45aa_baf0_933740e18bf5.slice/crio-e6cb9fc835220189b7fb3c11d9bfe6509279189a43773cd687ea5379ea82f7a4 WatchSource:0}: Error finding container e6cb9fc835220189b7fb3c11d9bfe6509279189a43773cd687ea5379ea82f7a4: Status 404 returned error can't find the container with id e6cb9fc835220189b7fb3c11d9bfe6509279189a43773cd687ea5379ea82f7a4
Oct 30 19:01:39.984060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:39.984016    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9" event={"ID":"cee017bb-6870-45aa-baf0-933740e18bf5","Type":"ContainerStarted","Data":"06c54dd55bea8776ae0c8d761cb3692e30df85fd448136d7bdbd8a567fdfb4b5"}
Oct 30 19:01:39.984509 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:39.984116    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9" event={"ID":"cee017bb-6870-45aa-baf0-933740e18bf5","Type":"ContainerStarted","Data":"e6cb9fc835220189b7fb3c11d9bfe6509279189a43773cd687ea5379ea82f7a4"}
Oct 30 19:01:40.349065 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:01:40.349017    4267 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee017bb_6870_45aa_baf0_933740e18bf5.slice/crio-06c54dd55bea8776ae0c8d761cb3692e30df85fd448136d7bdbd8a567fdfb4b5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee017bb_6870_45aa_baf0_933740e18bf5.slice/crio-conmon-06c54dd55bea8776ae0c8d761cb3692e30df85fd448136d7bdbd8a567fdfb4b5.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:01:40.417244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:40.417210    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:01:40.417339 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:40.417275    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:01:40.994678 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:40.994593    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq" event={"ID":"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908","Type":"ContainerStarted","Data":"40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45"}
Oct 30 19:01:40.995088 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:40.994819    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:40.996389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:40.996363    4267 generic.go:356] "Generic (PLEG): container finished" podID="cee017bb-6870-45aa-baf0-933740e18bf5" containerID="06c54dd55bea8776ae0c8d761cb3692e30df85fd448136d7bdbd8a567fdfb4b5" exitCode=0
Oct 30 19:01:40.996533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:40.996440    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9" event={"ID":"cee017bb-6870-45aa-baf0-933740e18bf5","Type":"ContainerDied","Data":"06c54dd55bea8776ae0c8d761cb3692e30df85fd448136d7bdbd8a567fdfb4b5"}
Oct 30 19:01:41.214907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:41.214851    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq" podStartSLOduration=2.345015681 podStartE2EDuration="7.214836887s" totalImagesPullingTime="4.869821206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:41.161243112 +0000 UTC m=+1392.049160202" watchObservedRunningTime="2026-10-30 19:01:41.214836887 +0000 UTC m=+1392.102753975"
Oct 30 19:01:41.215123 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:41.214976    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9" podStartSLOduration=3.214969585 podStartE2EDuration="3.214969585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:40.102334238 +0000 UTC m=+1390.990251342" watchObservedRunningTime="2026-10-30 19:01:41.214969585 +0000 UTC m=+1392.102886676"
Oct 30 19:01:42.008204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.008172    4267 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jdgbq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:01:42.008620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.008233    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq" podUID="b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:01:42.196581 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.196553    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:42.284857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.284773    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee017bb-6870-45aa-baf0-933740e18bf5-kube-api-access-cls9r\" (UniqueName: \"kubernetes.io/projected/cee017bb-6870-45aa-baf0-933740e18bf5-kube-api-access-cls9r\") pod \"cee017bb-6870-45aa-baf0-933740e18bf5\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") "
Oct 30 19:01:42.284857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.284822    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-xks-ca\") pod \"cee017bb-6870-45aa-baf0-933740e18bf5\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") "
Oct 30 19:01:42.284857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.284847    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-openshift-ca\") pod \"cee017bb-6870-45aa-baf0-933740e18bf5\" (UID: \"cee017bb-6870-45aa-baf0-933740e18bf5\") "
Oct 30 19:01:42.285266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.285233    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-openshift-ca" pod "cee017bb-6870-45aa-baf0-933740e18bf5" (UID: "cee017bb-6870-45aa-baf0-933740e18bf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:42.285673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.285654    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:42.287572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.287546    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-xks-ca" pod "cee017bb-6870-45aa-baf0-933740e18bf5" (UID: "cee017bb-6870-45aa-baf0-933740e18bf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:42.287637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.287581    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee017bb-6870-45aa-baf0-933740e18bf5-kube-api-access-cls9r" pod "cee017bb-6870-45aa-baf0-933740e18bf5" (UID: "cee017bb-6870-45aa-baf0-933740e18bf5"). InnerVolumeSpecName "kube-api-access-cls9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:42.387125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.387090    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cls9r\" (UniqueName: \"kubernetes.io/projected/cee017bb-6870-45aa-baf0-933740e18bf5-kube-api-access-cls9r\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:42.387125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:42.387120    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee017bb-6870-45aa-baf0-933740e18bf5-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:01:43.016668 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:43.016632    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"
Oct 30 19:01:43.017047 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:43.016668    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9" event={"ID":"cee017bb-6870-45aa-baf0-933740e18bf5","Type":"ContainerDied","Data":"e6cb9fc835220189b7fb3c11d9bfe6509279189a43773cd687ea5379ea82f7a4"}
Oct 30 19:01:43.017047 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:43.016731    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6cb9fc835220189b7fb3c11d9bfe6509279189a43773cd687ea5379ea82f7a4"
Oct 30 19:01:45.006893 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:45.006860    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:01:45.053456 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:45.053421    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"]
Oct 30 19:01:45.102474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:45.102441    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a21455-rc7hs"]
Oct 30 19:01:45.224827 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:45.224781    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defae603-d594-4eb2-b349-cc30ace02f2b" path="/var/lib/kubelet/pods/defae603-d594-4eb2-b349-cc30ace02f2b/volumes"
Oct 30 19:01:48.771623 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:48.771579    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.850726 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:48.850658    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.855642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:48.855570    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.875689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:48.875634    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.878273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:48.876974    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.900708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:48.900647    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:01:48.901059 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:48.901029    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.088202 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.088097    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.088763 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.088100    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.240523 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.237835    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.300519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.292589    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.327643 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.327589    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.403587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.402740    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:01:49.593518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.570634    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.814522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.780270    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:01:49.814522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.782097    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:01:49.882529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.879669    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:49.893528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.886583    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:01:49.958519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.923640    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:01:49.958519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.925326    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:01:49.958519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.925392    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:01:49.958519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.925431    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:01:49.958519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.939220    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:01:50.054607 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.964574    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.054607 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.969946    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:01:50.054607 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.971612    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:01:50.054607 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.971685    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:01:50.054607 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:49.971726    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:01:50.138646 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.091609    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.138646 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.097580    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:01:50.138646 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.129993    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.138646 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.131383    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:01:50.175538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.168514    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 30 19:01:50.255350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.240236    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.346562 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.334618    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.423470 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.404028    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.424401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.424374    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:01:50.447548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.435601    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:01:50.447548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.435912    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:01:50.819519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.804209    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:01:50.931559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.909382    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.931559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.909754    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:01:50.932067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.932024    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:01:50.990519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.981383    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.991306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.991258    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:01:50.999526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.994573    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:50.999526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:50.994891    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded"
Oct 30 19:01:51.097691 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:51.097578    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:51.157837 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:51.131373    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:01:51.279566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:51.250286    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:01:51.444576 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:51.435181    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:51.444576 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:51.437612    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vnbwf" podUID="acd523a4-65be-45f7-a79a-5c642c8f4787" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.19:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:51.469518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:51.461065    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:51.653518 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:01:51.602540    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:01:51.821584 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:51.815717    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.049469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.040989    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.049469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.041773    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.049469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.045814    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.081571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.051739    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.107520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.082802    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.116571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.080297    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.486878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.105051    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.487190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.293963    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.487367 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.461857    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.500520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.488114    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.560443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.560391    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.576518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.572636    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:01:52.597518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.590515    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.689945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.689898    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:52.819577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:52.816582    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.053593 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.050585    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.071914 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.071807    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.089790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.089736    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:01:53.123701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.123636    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.161662 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.161611    4267 patch_prober.go:29] interesting pod/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:53.173521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.165571    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.496521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.495589    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.496521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.496028    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.500515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.498814    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:01:53.562627 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.562577    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.562991 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.562965    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:01:53.573633 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.573587    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.607528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.603588    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.822112 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.821994    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:53.945874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.945825    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:53.946191 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:53.946143    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.061522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.056611    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:01:54.092695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.092591    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.093333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.093300    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:01:54.134456 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.133612    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:01:54.583516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.582078    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.583516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.582451    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:01:54.583516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.582583    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.583516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.582766    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.583516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.582902    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:01:54.678532 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.677722    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:54.702518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.683980    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:01:54.854997 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:54.850778    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:55.002271 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.002123    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:01:55.003380 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.003359    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:01:55.027678 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.027647    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:01:55.087190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.087147    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:55.105195 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.105105    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"]
Oct 30 19:01:55.110517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.110452    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff8cc94-5wxnm"]
Oct 30 19:01:55.120878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.120836    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:55.130312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.130078    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:01:55.386198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.386118    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee12d538-c02d-4c75-a2f7-1a654aaef531" path="/var/lib/kubelet/pods/ee12d538-c02d-4c75-a2f7-1a654aaef531/volumes"
Oct 30 19:01:55.386961 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.386936    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:01:55.387182 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.387167    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:01:55.387329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.387316    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:01:55.476361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.476318    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:01:55.482741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.482703    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:01:55.486507 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.485561    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:01:55.486507 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.485619    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:01:55.540270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.534037    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:01:55.540270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:55.534446    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:01:59.696803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.696766    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"]
Oct 30 19:01:59.697357 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.697342    4267 cpu_manager.go:414] "containerMap: removing container" podUID="cee017bb-6870-45aa-baf0-933740e18bf5" containerName="revoke-keys"
Oct 30 19:01:59.697400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.697358    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee017bb-6870-45aa-baf0-933740e18bf5" containerName="revoke-keys"
Oct 30 19:01:59.697445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.697437    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee017bb-6870-45aa-baf0-933740e18bf5" containerName="revoke-keys"
Oct 30 19:01:59.701970 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.701954    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:01:59.755412 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.755376    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-098a255d-zthg7\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:01:59.755658 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.755447    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg85x\" (UniqueName: \"kubernetes.io/projected/9be3e767-d8fc-4301-a64b-76c2c360f428-kube-api-access-wg85x\") pod \"maas-api-revoke-sub-098a255d-zthg7\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:01:59.755658 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.755521    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-098a255d-zthg7\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:01:59.804705 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.804668    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:01:59.804705 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.804668    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:59.809812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.809759    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"]
Oct 30 19:01:59.857066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.857034    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-098a255d-zthg7\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:01:59.857244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.857080    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg85x\" (UniqueName: \"kubernetes.io/projected/9be3e767-d8fc-4301-a64b-76c2c360f428-kube-api-access-wg85x\") pod \"maas-api-revoke-sub-098a255d-zthg7\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:01:59.857244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.857117    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-098a255d-zthg7\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:01:59.857742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.857717    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-098a255d-zthg7\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:01:59.859472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.859447    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-098a255d-zthg7\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:01:59.915110 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:01:59.915059    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg85x\" (UniqueName: \"kubernetes.io/projected/9be3e767-d8fc-4301-a64b-76c2c360f428-kube-api-access-wg85x\") pod \"maas-api-revoke-sub-098a255d-zthg7\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:02:00.017510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:00.017410    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:02:00.401129 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:02:00.401098    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be3e767_d8fc_4301_a64b_76c2c360f428.slice/crio-06b15ed6bde1aba29d01820d23dc5253cb0c90ad14fb815e0be9daa65d747326 WatchSource:0}: Error finding container 06b15ed6bde1aba29d01820d23dc5253cb0c90ad14fb815e0be9daa65d747326: Status 404 returned error can't find the container with id 06b15ed6bde1aba29d01820d23dc5253cb0c90ad14fb815e0be9daa65d747326
Oct 30 19:02:00.403067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:00.403041    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"]
Oct 30 19:02:01.084725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:01.084630    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7" event={"ID":"9be3e767-d8fc-4301-a64b-76c2c360f428","Type":"ContainerStarted","Data":"07efb2bfd30269da943212fb7c42539b444566b9ce1f4d80c1c72245e17be512"}
Oct 30 19:02:01.084725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:01.084687    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7" event={"ID":"9be3e767-d8fc-4301-a64b-76c2c360f428","Type":"ContainerStarted","Data":"06b15ed6bde1aba29d01820d23dc5253cb0c90ad14fb815e0be9daa65d747326"}
Oct 30 19:02:01.203428 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:01.203369    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7" podStartSLOduration=2.203347107 podStartE2EDuration="2.203347107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:01.199854138 +0000 UTC m=+1412.087771228" watchObservedRunningTime="2026-10-30 19:02:01.203347107 +0000 UTC m=+1412.091264196"
Oct 30 19:02:02.093619 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:02.093579    4267 generic.go:356] "Generic (PLEG): container finished" podID="9be3e767-d8fc-4301-a64b-76c2c360f428" containerID="07efb2bfd30269da943212fb7c42539b444566b9ce1f4d80c1c72245e17be512" exitCode=0
Oct 30 19:02:02.094024 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:02.093665    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7" event={"ID":"9be3e767-d8fc-4301-a64b-76c2c360f428","Type":"ContainerDied","Data":"07efb2bfd30269da943212fb7c42539b444566b9ce1f4d80c1c72245e17be512"}
Oct 30 19:02:03.265892 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:03.265865    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:02:03.308540 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:03.308474    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-xks-ca\") pod \"9be3e767-d8fc-4301-a64b-76c2c360f428\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") "
Oct 30 19:02:03.308735 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:03.308706    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be3e767-d8fc-4301-a64b-76c2c360f428-kube-api-access-wg85x\" (UniqueName: \"kubernetes.io/projected/9be3e767-d8fc-4301-a64b-76c2c360f428-kube-api-access-wg85x\") pod \"9be3e767-d8fc-4301-a64b-76c2c360f428\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") "
Oct 30 19:02:03.308938 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:03.308923    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-openshift-ca\") pod \"9be3e767-d8fc-4301-a64b-76c2c360f428\" (UID: \"9be3e767-d8fc-4301-a64b-76c2c360f428\") "
Oct 30 19:02:03.309270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:03.309245    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-openshift-ca" pod "9be3e767-d8fc-4301-a64b-76c2c360f428" (UID: "9be3e767-d8fc-4301-a64b-76c2c360f428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:03.309586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:03.309568    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:03.310670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:03.310651    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-xks-ca" pod "9be3e767-d8fc-4301-a64b-76c2c360f428" (UID: "9be3e767-d8fc-4301-a64b-76c2c360f428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:03.310855 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:03.310835    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be3e767-d8fc-4301-a64b-76c2c360f428-kube-api-access-wg85x" pod "9be3e767-d8fc-4301-a64b-76c2c360f428" (UID: "9be3e767-d8fc-4301-a64b-76c2c360f428"). InnerVolumeSpecName "kube-api-access-wg85x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:03.411382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:03.411283    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9be3e767-d8fc-4301-a64b-76c2c360f428-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:03.411382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:03.411327    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg85x\" (UniqueName: \"kubernetes.io/projected/9be3e767-d8fc-4301-a64b-76c2c360f428-kube-api-access-wg85x\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:04.111778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.111741    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"
Oct 30 19:02:04.111778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.111757    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7" event={"ID":"9be3e767-d8fc-4301-a64b-76c2c360f428","Type":"ContainerDied","Data":"06b15ed6bde1aba29d01820d23dc5253cb0c90ad14fb815e0be9daa65d747326"}
Oct 30 19:02:04.112006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.111807    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06b15ed6bde1aba29d01820d23dc5253cb0c90ad14fb815e0be9daa65d747326"
Oct 30 19:02:04.314414 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.314379    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"]
Oct 30 19:02:04.315028 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.315012    4267 cpu_manager.go:414] "containerMap: removing container" podUID="9be3e767-d8fc-4301-a64b-76c2c360f428" containerName="revoke-keys"
Oct 30 19:02:04.315074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.315030    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be3e767-d8fc-4301-a64b-76c2c360f428" containerName="revoke-keys"
Oct 30 19:02:04.315161 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.315152    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be3e767-d8fc-4301-a64b-76c2c360f428" containerName="revoke-keys"
Oct 30 19:02:04.318806 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.318784    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.421854 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.421763    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"]
Oct 30 19:02:04.424581 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.424552    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.424723 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.424611    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.424723 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.424705    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.424849 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.424819    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d32d01-45a7-4b42-be96-e61fe8ae3545-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.425213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.424921    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.425213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.425066    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpkw6\" (UniqueName: \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-kube-api-access-rpkw6\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.425213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.425140    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.425213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.425164    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.425430 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.425239    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.526575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.526534    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.526575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.526577    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.526794 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.526639    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d32d01-45a7-4b42-be96-e61fe8ae3545-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.526794 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.526720    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.526887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.526810    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpkw6\" (UniqueName: \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-kube-api-access-rpkw6\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.526945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.526925    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.526986 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.526967    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.527074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.526981    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-data\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.527074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.527047    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.527186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.527116    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.527295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.527273    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-certs\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.527586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.527558    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d32d01-45a7-4b42-be96-e61fe8ae3545-istiod-ca-cert\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.527586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.527569    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.527777 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.527557    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-credential-socket\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.529399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.529370    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-envoy\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.529647 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.529628    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-podinfo\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.583328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.583294    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-token\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.583560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.583528    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpkw6\" (UniqueName: \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-kube-api-access-rpkw6\") pod \"e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") " pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:04.632668 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:04.632628    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:05.077611 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:05.077580    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"]
Oct 30 19:02:05.077742 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:02:05.077718    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d32d01_45a7_4b42_be96_e61fe8ae3545.slice/crio-e879715d673b8903cd9f949e1ffaa522e59c62107a24a799d32684e3b46c5fc3 WatchSource:0}: Error finding container e879715d673b8903cd9f949e1ffaa522e59c62107a24a799d32684e3b46c5fc3: Status 404 returned error can't find the container with id e879715d673b8903cd9f949e1ffaa522e59c62107a24a799d32684e3b46c5fc3
Oct 30 19:02:05.080453 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:05.080418    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:02:05.080576 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:05.080488    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:02:05.080576 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:05.080541    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:02:05.121957 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:05.121914    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" event={"ID":"80d32d01-45a7-4b42-be96-e61fe8ae3545","Type":"ContainerStarted","Data":"e879715d673b8903cd9f949e1ffaa522e59c62107a24a799d32684e3b46c5fc3"}
Oct 30 19:02:06.141829 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:06.141793    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" event={"ID":"80d32d01-45a7-4b42-be96-e61fe8ae3545","Type":"ContainerStarted","Data":"9491b161f2b0f9d7722191c06ecc70b68a7ef5bf76eab3177fb82971ea58a485"}
Oct 30 19:02:06.335517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:06.334670    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" podStartSLOduration=2.33464938 podStartE2EDuration="2.33464938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:06.330768534 +0000 UTC m=+1417.218685624" watchObservedRunningTime="2026-10-30 19:02:06.33464938 +0000 UTC m=+1417.222566470"
Oct 30 19:02:06.635273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:06.634869    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:06.636690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:06.636660    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:02:06.636811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:06.636714    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:02:07.633938 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:07.633902    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:02:07.634444 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:07.633957    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:02:08.634096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:08.634053    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:02:08.634567 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:08.634126    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:02:09.634028 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:09.633985    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:02:09.634257 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:09.634060    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:02:10.362236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.362204    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"]
Oct 30 19:02:10.366717 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.366698    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:10.445968 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.445925    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25m75\" (UniqueName: \"kubernetes.io/projected/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-kube-api-access-25m75\") pod \"maas-api-revoke-sub-797c813d-2kgqg\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:10.446284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.446262    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-797c813d-2kgqg\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:10.446513 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.446478    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-797c813d-2kgqg\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:10.447671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.447642    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"]
Oct 30 19:02:10.469623 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.469593    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:10.469817 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.469734    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:02:10.548324 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.548287    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25m75\" (UniqueName: \"kubernetes.io/projected/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-kube-api-access-25m75\") pod \"maas-api-revoke-sub-797c813d-2kgqg\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:10.548571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.548357    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-797c813d-2kgqg\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:10.548571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.548401    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-797c813d-2kgqg\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:10.549172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.549145    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-797c813d-2kgqg\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:10.550815 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.550790    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-797c813d-2kgqg\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:10.605259 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.605228    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25m75\" (UniqueName: \"kubernetes.io/projected/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-kube-api-access-25m75\") pod \"maas-api-revoke-sub-797c813d-2kgqg\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:10.637434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.636954    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:02:10.637434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.637016    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:02:10.680833 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:10.680794    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:11.094696 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:11.094665    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"]
Oct 30 19:02:11.095111 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:02:11.095084    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62447b2_8fb9_4f32_a8cb_1e8a57bd2ea5.slice/crio-6f6e9941cfbc6a631b0c0610f830aef8f8dd34bc45f092f243888c38b9c9162a WatchSource:0}: Error finding container 6f6e9941cfbc6a631b0c0610f830aef8f8dd34bc45f092f243888c38b9c9162a: Status 404 returned error can't find the container with id 6f6e9941cfbc6a631b0c0610f830aef8f8dd34bc45f092f243888c38b9c9162a
Oct 30 19:02:11.187944 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:11.187907    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg" event={"ID":"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5","Type":"ContainerStarted","Data":"6f6e9941cfbc6a631b0c0610f830aef8f8dd34bc45f092f243888c38b9c9162a"}
Oct 30 19:02:11.634121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:11.634083    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 19:02:11.634333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:11.634166    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 19:02:12.199012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:12.198963    4267 generic.go:356] "Generic (PLEG): container finished" podID="a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5" containerID="5e7be18a1580595983eb1f5988cbf887f35702c6f2980331a11cdec15d391a5e" exitCode=0
Oct 30 19:02:12.199549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:12.199040    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg" event={"ID":"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5","Type":"ContainerDied","Data":"5e7be18a1580595983eb1f5988cbf887f35702c6f2980331a11cdec15d391a5e"}
Oct 30 19:02:12.643088 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:12.643053    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:12.643621 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:12.643596    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:12.644469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:12.644449    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:02:13.415185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:13.415157    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:13.601647 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:13.601563    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-kube-api-access-25m75\" (UniqueName: \"kubernetes.io/projected/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-kube-api-access-25m75\") pod \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") "
Oct 30 19:02:13.601893 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:13.601875    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-openshift-ca\") pod \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") "
Oct 30 19:02:13.601952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:13.601906    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-xks-ca\") pod \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\" (UID: \"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5\") "
Oct 30 19:02:13.602297 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:13.602267    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-openshift-ca" pod "a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5" (UID: "a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:13.602434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:13.602420    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:13.603928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:13.603901    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-kube-api-access-25m75" pod "a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5" (UID: "a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5"). InnerVolumeSpecName "kube-api-access-25m75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:13.604061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:13.604036    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-xks-ca" pod "a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5" (UID: "a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:13.703857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:13.703819    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:13.703857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:13.703853    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25m75\" (UniqueName: \"kubernetes.io/projected/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5-kube-api-access-25m75\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:14.230587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:14.230555    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"
Oct 30 19:02:14.230587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:14.230568    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg" event={"ID":"a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5","Type":"ContainerDied","Data":"6f6e9941cfbc6a631b0c0610f830aef8f8dd34bc45f092f243888c38b9c9162a"}
Oct 30 19:02:14.230835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:14.230620    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f6e9941cfbc6a631b0c0610f830aef8f8dd34bc45f092f243888c38b9c9162a"
Oct 30 19:02:20.161558 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:02:20.161510    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:02:22.678641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:22.678587    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:22.825652 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:22.825596    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:22.928473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:22.926808    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:02:22.938641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:22.938592    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:22.991649 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:22.991592    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.065630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.065581    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.107658 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.107600    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.475587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.470588    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.475933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.475894    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:02:23.514639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.514583    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:23.515766 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.515732    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:02:23.905700 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.905650    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:02:24.046519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.927149    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.046519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.962740    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:02:24.046519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.962801    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:02:24.046519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.961158    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.046519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.961199    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:02:24.046519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:23.976818    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:02:24.046519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.004604    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.046519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.005061    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:02:24.071518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.060635    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.096520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.073627    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.096520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.073890    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:02:24.232518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.217073    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.259518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.232892    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:02:24.502077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.473611    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.552659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.532774    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:02:24.552659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.480613    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.552659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.522960    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.552659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.541665    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:02:24.552659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.541736    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:02:24.552659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.522923    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:24.552659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.549106    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:02:24.987706 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.987658    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.088071 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:24.991585    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.095588 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.095542    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:02:25.096253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.096215    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:02:25.121910 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.121871    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.265346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.122268    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.266634 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.058202    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.281627 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.130603    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:02:25.281627 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.267627    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:02:25.295655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.295605    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.580567 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.536182    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.602226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.540748    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.640515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.540794    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.641105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.553878    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:25.720688 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.720648    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 19:02:25.834693 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:25.834579    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 30 19:02:26.092695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.092592    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:02:26.415571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.106361    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.415571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.394863    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.415571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.140589    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.497520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.144328    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:02:26.497520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.390161    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.661517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.144212    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.723301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.611772    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.724006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.612002    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.836744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.612131    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:02:26.844647 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.630016    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:02:26.940810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.907133    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:26.958908 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:26.958862    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.132421 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.132373    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.151075 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.139801    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:02:27.151433 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.151400    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:02:27.408194 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.399752    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.508481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.508438    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.510363 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.510331    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:02:27.511078 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.511056    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:02:27.521133 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.520651    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:02:27.521133 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.520695    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.573906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.573862    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.640523 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.638833    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:02:27.671452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.671352    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:02:27.712515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.692659    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:02:27.721722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.721691    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:02:27.724352 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.724319    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:27.727512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.727301    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:02:27.727913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.727890    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:02:27.772376 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.769641    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:02:27.779673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.779649    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:02:27.792938 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.792915    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:02:27.798324 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:27.798303    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:02:29.421445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:29.421403    4267 scope.go:122] "RemoveContainer" containerID="781390455e6125d0b933a8530d9c13fb3064251cbd564075d355093b5568d4ef"
Oct 30 19:02:29.437946 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:29.437921    4267 scope.go:122] "RemoveContainer" containerID="4f442c342489bdeecc3d1b7e43d5ee65a7038a28d29bdbb4cc4089395b184142"
Oct 30 19:02:29.451842 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:29.451802    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"]
Oct 30 19:02:29.452079 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:29.452053    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerName="istio-proxy" containerID="cri-o://e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af" gracePeriod=30
Oct 30 19:02:32.073772 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.073731    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"]
Oct 30 19:02:32.074524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.074486    4267 cpu_manager.go:414] "containerMap: removing container" podUID="a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5" containerName="revoke-keys"
Oct 30 19:02:32.074632 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.074525    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5" containerName="revoke-keys"
Oct 30 19:02:32.074735 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.074721    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5" containerName="revoke-keys"
Oct 30 19:02:32.079856 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.079834    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.112510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.112446    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f9e7b1-9w4bf\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.113393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.112768    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfbw\" (UniqueName: \"kubernetes.io/projected/20c5dc0d-17ca-4227-b009-cf8fca9cd182-kube-api-access-qlfbw\") pod \"maas-api-revoke-sub-84f9e7b1-9w4bf\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.113393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.112877    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f9e7b1-9w4bf\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.135550 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.135524    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"]
Oct 30 19:02:32.177581 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.177532    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:32.177782 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.177659    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:02:32.214787 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.214756    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f9e7b1-9w4bf\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.214933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.214827    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f9e7b1-9w4bf\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.215029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.215011    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfbw\" (UniqueName: \"kubernetes.io/projected/20c5dc0d-17ca-4227-b009-cf8fca9cd182-kube-api-access-qlfbw\") pod \"maas-api-revoke-sub-84f9e7b1-9w4bf\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.215741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.215715    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f9e7b1-9w4bf\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.217242 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.217215    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f9e7b1-9w4bf\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.271826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.271792    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfbw\" (UniqueName: \"kubernetes.io/projected/20c5dc0d-17ca-4227-b009-cf8fca9cd182-kube-api-access-qlfbw\") pod \"maas-api-revoke-sub-84f9e7b1-9w4bf\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.393638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.393552    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:32.815969 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:02:32.815932    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c5dc0d_17ca_4227_b009_cf8fca9cd182.slice/crio-e2c00aff59b90217449db4c78af7b797ddb2afb402507174eaba0a0c1249dcae WatchSource:0}: Error finding container e2c00aff59b90217449db4c78af7b797ddb2afb402507174eaba0a0c1249dcae: Status 404 returned error can't find the container with id e2c00aff59b90217449db4c78af7b797ddb2afb402507174eaba0a0c1249dcae
Oct 30 19:02:32.817564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:32.817536    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"]
Oct 30 19:02:33.446572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:33.446533    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf" event={"ID":"20c5dc0d-17ca-4227-b009-cf8fca9cd182","Type":"ContainerStarted","Data":"d38cd835ec59ddcdbdf62e1e73014f85d3458b6747059b1892e51341c4af5315"}
Oct 30 19:02:33.446967 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:33.446592    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf" event={"ID":"20c5dc0d-17ca-4227-b009-cf8fca9cd182","Type":"ContainerStarted","Data":"e2c00aff59b90217449db4c78af7b797ddb2afb402507174eaba0a0c1249dcae"}
Oct 30 19:02:33.560540 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:33.560469    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf" podStartSLOduration=1.56045458 podStartE2EDuration="1.56045458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:33.559277889 +0000 UTC m=+1444.447194982" watchObservedRunningTime="2026-10-30 19:02:33.56045458 +0000 UTC m=+1444.448371669"
Oct 30 19:02:34.455383 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.455348    4267 generic.go:356] "Generic (PLEG): container finished" podID="20c5dc0d-17ca-4227-b009-cf8fca9cd182" containerID="d38cd835ec59ddcdbdf62e1e73014f85d3458b6747059b1892e51341c4af5315" exitCode=0
Oct 30 19:02:34.456030 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.455439    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf" event={"ID":"20c5dc0d-17ca-4227-b009-cf8fca9cd182","Type":"ContainerDied","Data":"d38cd835ec59ddcdbdf62e1e73014f85d3458b6747059b1892e51341c4af5315"}
Oct 30 19:02:34.730669 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.730645    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 19:02:34.862970 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.862936    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-certs\") pod \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") "
Oct 30 19:02:34.863163 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863001    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-token\" (UniqueName: \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-token\") pod \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") "
Oct 30 19:02:34.863163 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863079    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-socket\") pod \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") "
Oct 30 19:02:34.863163 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863130    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-kube-api-access-484zd\" (UniqueName: \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-kube-api-access-484zd\") pod \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") "
Oct 30 19:02:34.863327 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863210    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-podinfo\") pod \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") "
Oct 30 19:02:34.863327 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863250    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-credential-socket\") pod \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") "
Oct 30 19:02:34.863431 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863343    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-envoy\") pod \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") "
Oct 30 19:02:34.863487 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863451    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istiod-ca-cert\") pod \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") "
Oct 30 19:02:34.863590 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863540    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-data\") pod \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\" (UID: \"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9\") "
Oct 30 19:02:34.863913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863242    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-certs" pod "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" (UID: "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:34.863913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863573    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-socket" pod "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" (UID: "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:34.863913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.863804    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-credential-socket" pod "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" (UID: "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:34.864172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.864146    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-data" pod "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" (UID: "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:34.864421 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.864394    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istiod-ca-cert" pod "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" (UID: "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:34.864809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.864778    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:34.864905 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.864815    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:34.864905 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.864830    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:34.864905 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.864843    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:34.864905 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.864858    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:34.866696 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.866671    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-envoy" pod "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" (UID: "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:34.867039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.867006    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-token" pod "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" (UID: "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:34.867301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.867275    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-podinfo" pod "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" (UID: "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:02:34.867428 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.867353    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-kube-api-access-484zd" pod "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" (UID: "b0bf2ab1-b51b-4d67-b722-1ff40f17eef9"). InnerVolumeSpecName "kube-api-access-484zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:34.967166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.967044    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:34.967166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.967082    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-484zd\" (UniqueName: \"kubernetes.io/projected/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-kube-api-access-484zd\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:34.967166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.967099    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:34.967166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:34.967113    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:35.464038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.463998    4267 generic.go:356] "Generic (PLEG): container finished" podID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerID="e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af" exitCode=0
Oct 30 19:02:35.464454 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.464056    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"
Oct 30 19:02:35.464454 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.464084    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" event={"ID":"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9","Type":"ContainerDied","Data":"e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af"}
Oct 30 19:02:35.464454 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.464169    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd" event={"ID":"b0bf2ab1-b51b-4d67-b722-1ff40f17eef9","Type":"ContainerDied","Data":"ce227e238919a59448746e3e9c2f5d67d2864fee076148d8958db25e7f4e6969"}
Oct 30 19:02:35.464454 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.464197    4267 scope.go:122] "RemoveContainer" containerID="e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af"
Oct 30 19:02:35.475717 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.475683    4267 scope.go:122] "RemoveContainer" containerID="e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af"
Oct 30 19:02:35.476076 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:02:35.476043    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af\": container with ID starting with e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af not found: ID does not exist" containerID="e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af"
Oct 30 19:02:35.476171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.476088    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af"} err="failed to get container status \"e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af\": rpc error: code = NotFound desc = could not find container \"e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af\": container with ID starting with e0b4c90c2e31ecd91e0ef9755bff49f9c5d574e4653917676f16d9fca57e76af not found: ID does not exist"
Oct 30 19:02:35.641952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.641918    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:35.693538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.687791    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"]
Oct 30 19:02:35.730930 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.730896    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7zrcd"]
Oct 30 19:02:35.788616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.781469    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-xks-ca\") pod \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") "
Oct 30 19:02:35.788616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.781569    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-openshift-ca\") pod \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") "
Oct 30 19:02:35.788616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.781666    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c5dc0d-17ca-4227-b009-cf8fca9cd182-kube-api-access-qlfbw\" (UniqueName: \"kubernetes.io/projected/20c5dc0d-17ca-4227-b009-cf8fca9cd182-kube-api-access-qlfbw\") pod \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\" (UID: \"20c5dc0d-17ca-4227-b009-cf8fca9cd182\") "
Oct 30 19:02:35.788616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.782154    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-openshift-ca" pod "20c5dc0d-17ca-4227-b009-cf8fca9cd182" (UID: "20c5dc0d-17ca-4227-b009-cf8fca9cd182"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:35.788616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.782402    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:35.788616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.784644    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-xks-ca" pod "20c5dc0d-17ca-4227-b009-cf8fca9cd182" (UID: "20c5dc0d-17ca-4227-b009-cf8fca9cd182"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:35.788616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.785625    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c5dc0d-17ca-4227-b009-cf8fca9cd182-kube-api-access-qlfbw" pod "20c5dc0d-17ca-4227-b009-cf8fca9cd182" (UID: "20c5dc0d-17ca-4227-b009-cf8fca9cd182"). InnerVolumeSpecName "kube-api-access-qlfbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:35.884343 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.884309    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlfbw\" (UniqueName: \"kubernetes.io/projected/20c5dc0d-17ca-4227-b009-cf8fca9cd182-kube-api-access-qlfbw\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:35.884343 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:35.884340    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c5dc0d-17ca-4227-b009-cf8fca9cd182-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:36.474638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:36.474506    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"
Oct 30 19:02:36.474638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:36.474535    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf" event={"ID":"20c5dc0d-17ca-4227-b009-cf8fca9cd182","Type":"ContainerDied","Data":"e2c00aff59b90217449db4c78af7b797ddb2afb402507174eaba0a0c1249dcae"}
Oct 30 19:02:36.474638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:36.474623    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2c00aff59b90217449db4c78af7b797ddb2afb402507174eaba0a0c1249dcae"
Oct 30 19:02:37.223504 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:37.223469    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" path="/var/lib/kubelet/pods/b0bf2ab1-b51b-4d67-b722-1ff40f17eef9/volumes"
Oct 30 19:02:40.114162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.114125    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"]
Oct 30 19:02:40.115187 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.115165    4267 cpu_manager.go:414] "containerMap: removing container" podUID="20c5dc0d-17ca-4227-b009-cf8fca9cd182" containerName="revoke-keys"
Oct 30 19:02:40.115319 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.115188    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c5dc0d-17ca-4227-b009-cf8fca9cd182" containerName="revoke-keys"
Oct 30 19:02:40.115319 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.115229    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerName="istio-proxy"
Oct 30 19:02:40.115319 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.115239    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerName="istio-proxy"
Oct 30 19:02:40.115538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.115382    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bf2ab1-b51b-4d67-b722-1ff40f17eef9" containerName="istio-proxy"
Oct 30 19:02:40.115538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.115397    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c5dc0d-17ca-4227-b009-cf8fca9cd182" containerName="revoke-keys"
Oct 30 19:02:40.127766 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.127736    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.204374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.204339    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"]
Oct 30 19:02:40.258114 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.258076    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a495d43-lfdkn\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.258328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.258152    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a495d43-lfdkn\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.258381 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.258345    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qf6d\" (UniqueName: \"kubernetes.io/projected/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-kube-api-access-4qf6d\") pod \"maas-api-revoke-sub-3a495d43-lfdkn\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.261371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.261353    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:40.281936 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.281902    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:02:40.360203 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.360165    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a495d43-lfdkn\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.360392 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.360292    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qf6d\" (UniqueName: \"kubernetes.io/projected/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-kube-api-access-4qf6d\") pod \"maas-api-revoke-sub-3a495d43-lfdkn\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.360545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.360527    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a495d43-lfdkn\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.361348 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.361322    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a495d43-lfdkn\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.362884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.362865    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a495d43-lfdkn\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.417650 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.417574    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qf6d\" (UniqueName: \"kubernetes.io/projected/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-kube-api-access-4qf6d\") pod \"maas-api-revoke-sub-3a495d43-lfdkn\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.459353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.459317    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:40.862968 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:40.862927    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"]
Oct 30 19:02:40.863354 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:02:40.863329    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f3ccd6_f8e9_47bb_aa0e_c839ef0ad9ee.slice/crio-df623ed5965124e0ab30d888959d07eaf31991d5f27bf74b7129fbc694daf7be WatchSource:0}: Error finding container df623ed5965124e0ab30d888959d07eaf31991d5f27bf74b7129fbc694daf7be: Status 404 returned error can't find the container with id df623ed5965124e0ab30d888959d07eaf31991d5f27bf74b7129fbc694daf7be
Oct 30 19:02:41.059522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:41.059471    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"]
Oct 30 19:02:41.063971 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:41.063944    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d277d1-88svd"]
Oct 30 19:02:41.222343 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:41.222304    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1cf202-2a0f-4012-9d41-95f65077333b" path="/var/lib/kubelet/pods/2a1cf202-2a0f-4012-9d41-95f65077333b/volumes"
Oct 30 19:02:41.544976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:41.544936    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn" event={"ID":"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee","Type":"ContainerStarted","Data":"1fb72a0c118fc4e07bc50753b1fd86bf1cb130c1d79ffd39c021af9c56edcce6"}
Oct 30 19:02:41.545226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:41.545000    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn" event={"ID":"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee","Type":"ContainerStarted","Data":"df623ed5965124e0ab30d888959d07eaf31991d5f27bf74b7129fbc694daf7be"}
Oct 30 19:02:41.660224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:41.660168    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn" podStartSLOduration=1.660154499 podStartE2EDuration="1.660154499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:41.658073535 +0000 UTC m=+1452.545990631" watchObservedRunningTime="2026-10-30 19:02:41.660154499 +0000 UTC m=+1452.548071588"
Oct 30 19:02:42.559095 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:42.559056    4267 generic.go:356] "Generic (PLEG): container finished" podID="a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee" containerID="1fb72a0c118fc4e07bc50753b1fd86bf1cb130c1d79ffd39c021af9c56edcce6" exitCode=0
Oct 30 19:02:42.559825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:42.559347    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn" event={"ID":"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee","Type":"ContainerDied","Data":"1fb72a0c118fc4e07bc50753b1fd86bf1cb130c1d79ffd39c021af9c56edcce6"}
Oct 30 19:02:42.980189 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:42.980153    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"]
Oct 30 19:02:42.984681 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:42.984662    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:02:43.020169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.020127    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a2d6fd-783b-4e85-b831-0ad295744988-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn\" (UID: \"c6a2d6fd-783b-4e85-b831-0ad295744988\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:02:43.020418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.020395    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642lk\" (UniqueName: \"kubernetes.io/projected/c6a2d6fd-783b-4e85-b831-0ad295744988-kube-api-access-642lk\") pod \"payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn\" (UID: \"c6a2d6fd-783b-4e85-b831-0ad295744988\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:02:43.038340 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.037155    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cfaf8925\""
Oct 30 19:02:43.044027 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.043992    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"]
Oct 30 19:02:43.090564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.090437    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"]
Oct 30 19:02:43.106781 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.103931    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:02:43.121727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.121697    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a2d6fd-783b-4e85-b831-0ad295744988-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn\" (UID: \"c6a2d6fd-783b-4e85-b831-0ad295744988\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:02:43.121906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.121852    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd\" (UID: \"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66\") " pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:02:43.122062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.122046    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642lk\" (UniqueName: \"kubernetes.io/projected/c6a2d6fd-783b-4e85-b831-0ad295744988-kube-api-access-642lk\") pod \"payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn\" (UID: \"c6a2d6fd-783b-4e85-b831-0ad295744988\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:02:43.122156 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.122140    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db8x9\" (UniqueName: \"kubernetes.io/projected/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-kube-api-access-db8x9\") pod \"payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd\" (UID: \"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66\") " pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:02:43.123542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.123185    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a2d6fd-783b-4e85-b831-0ad295744988-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn\" (UID: \"c6a2d6fd-783b-4e85-b831-0ad295744988\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:02:43.153190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.153150    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"]
Oct 30 19:02:43.163303 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.163274    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cfaf8925-dockercfg-dxnn9\""
Oct 30 19:02:43.204540 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.204453    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-642lk\" (UniqueName: \"kubernetes.io/projected/c6a2d6fd-783b-4e85-b831-0ad295744988-kube-api-access-642lk\") pod \"payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn\" (UID: \"c6a2d6fd-783b-4e85-b831-0ad295744988\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:02:43.237571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.223234    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd\" (UID: \"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66\") " pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:02:43.237571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.223460    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db8x9\" (UniqueName: \"kubernetes.io/projected/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-kube-api-access-db8x9\") pod \"payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd\" (UID: \"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66\") " pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:02:43.237571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.225544    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd\" (UID: \"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66\") " pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:02:43.287215 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.286959    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db8x9\" (UniqueName: \"kubernetes.io/projected/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-kube-api-access-db8x9\") pod \"payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd\" (UID: \"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66\") " pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:02:43.309612 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.309531    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:02:43.435147 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.433153    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:02:43.715368 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:02:43.715331    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a2d6fd_783b_4e85_b831_0ad295744988.slice/crio-5684e4a5f7f5a90d28e0a8f72455b12cbe1001d02900c0dcffc34b98541e978f WatchSource:0}: Error finding container 5684e4a5f7f5a90d28e0a8f72455b12cbe1001d02900c0dcffc34b98541e978f: Status 404 returned error can't find the container with id 5684e4a5f7f5a90d28e0a8f72455b12cbe1001d02900c0dcffc34b98541e978f
Oct 30 19:02:43.717397 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.717371    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"]
Oct 30 19:02:43.732952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.732924    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:43.739873 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.739850    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-xks-ca\") pod \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") "
Oct 30 19:02:43.739991 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.739972    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-openshift-ca\") pod \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") "
Oct 30 19:02:43.740217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.740195    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-kube-api-access-4qf6d\" (UniqueName: \"kubernetes.io/projected/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-kube-api-access-4qf6d\") pod \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\" (UID: \"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee\") "
Oct 30 19:02:43.740314 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.740293    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-openshift-ca" pod "a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee" (UID: "a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:43.741124 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.740891    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:43.742230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.742204    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-xks-ca" pod "a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee" (UID: "a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:43.742825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.742795    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-kube-api-access-4qf6d" pod "a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee" (UID: "a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee"). InnerVolumeSpecName "kube-api-access-4qf6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:43.842371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.842334    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:43.842371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.842367    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qf6d\" (UniqueName: \"kubernetes.io/projected/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee-kube-api-access-4qf6d\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:02:43.846075 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:43.846047    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"]
Oct 30 19:02:43.847737 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:02:43.847703    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa5cbec_50c3_45a0_a6e0_8cfebbbc8d66.slice/crio-6ecb5aac1ce24a434500b523d0df05fb215565bd3925de215ca53c6a793ac168 WatchSource:0}: Error finding container 6ecb5aac1ce24a434500b523d0df05fb215565bd3925de215ca53c6a793ac168: Status 404 returned error can't find the container with id 6ecb5aac1ce24a434500b523d0df05fb215565bd3925de215ca53c6a793ac168
Oct 30 19:02:44.583138 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.583039    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd" event={"ID":"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66","Type":"ContainerStarted","Data":"c6006eb49f00b8f069b92f0a77307d639ff7ad1dc16d68738c8cc8d0e7ecbb53"}
Oct 30 19:02:44.583138 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.583107    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd" event={"ID":"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66","Type":"ContainerStarted","Data":"6ecb5aac1ce24a434500b523d0df05fb215565bd3925de215ca53c6a793ac168"}
Oct 30 19:02:44.583138 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.583136    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:02:44.585029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.584995    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn" event={"ID":"c6a2d6fd-783b-4e85-b831-0ad295744988","Type":"ContainerStarted","Data":"d027eadc3c7bfbe1d14004061e72e5e80753c2ca290ccb53984e1d829ada50ac"}
Oct 30 19:02:44.585184 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.585059    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:02:44.585184 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.585084    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn" event={"ID":"c6a2d6fd-783b-4e85-b831-0ad295744988","Type":"ContainerStarted","Data":"5684e4a5f7f5a90d28e0a8f72455b12cbe1001d02900c0dcffc34b98541e978f"}
Oct 30 19:02:44.586581 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.586557    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn" event={"ID":"a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee","Type":"ContainerDied","Data":"df623ed5965124e0ab30d888959d07eaf31991d5f27bf74b7129fbc694daf7be"}
Oct 30 19:02:44.586683 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.586598    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df623ed5965124e0ab30d888959d07eaf31991d5f27bf74b7129fbc694daf7be"
Oct 30 19:02:44.586683 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.586609    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"
Oct 30 19:02:44.705152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.705098    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd" podStartSLOduration=1.705084266 podStartE2EDuration="1.705084266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:44.699202313 +0000 UTC m=+1455.587119405" watchObservedRunningTime="2026-10-30 19:02:44.705084266 +0000 UTC m=+1455.593001356"
Oct 30 19:02:44.812617 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:44.812564    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn" podStartSLOduration=2.812550018 podStartE2EDuration="2.812550018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:44.810024483 +0000 UTC m=+1455.697941595" watchObservedRunningTime="2026-10-30 19:02:44.812550018 +0000 UTC m=+1455.700467107"
Oct 30 19:02:45.060544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:45.060477    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"]
Oct 30 19:02:45.065363 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:45.065333    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d7d05d-g8lct"]
Oct 30 19:02:45.222337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:45.222303    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c757c1-1c84-46d2-9b95-40f0942d766d" path="/var/lib/kubelet/pods/38c757c1-1c84-46d2-9b95-40f0942d766d/volumes"
Oct 30 19:02:50.161179 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:02:50.161137    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:02:52.731175 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.731139    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"]
Oct 30 19:02:52.731835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.731816    4267 cpu_manager.go:414] "containerMap: removing container" podUID="a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee" containerName="revoke-keys"
Oct 30 19:02:52.731890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.731837    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee" containerName="revoke-keys"
Oct 30 19:02:52.731959 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.731948    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee" containerName="revoke-keys"
Oct 30 19:02:52.735503 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.735477    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:02:52.760024 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.759986    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5988ce-d59e-4cb6-a637-997e33ba4541-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2\" (UID: \"8b5988ce-d59e-4cb6-a637-997e33ba4541\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:02:52.760175 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.760109    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpfc6\" (UniqueName: \"kubernetes.io/projected/8b5988ce-d59e-4cb6-a637-997e33ba4541-kube-api-access-cpfc6\") pod \"payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2\" (UID: \"8b5988ce-d59e-4cb6-a637-997e33ba4541\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:02:52.792755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.792705    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"]
Oct 30 19:02:52.865522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.865434    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5988ce-d59e-4cb6-a637-997e33ba4541-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2\" (UID: \"8b5988ce-d59e-4cb6-a637-997e33ba4541\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:02:52.865731 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.865540    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfc6\" (UniqueName: \"kubernetes.io/projected/8b5988ce-d59e-4cb6-a637-997e33ba4541-kube-api-access-cpfc6\") pod \"payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2\" (UID: \"8b5988ce-d59e-4cb6-a637-997e33ba4541\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:02:52.915521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.904663    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4b66bd06\""
Oct 30 19:02:52.945535 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.916850    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4b66bd06-dockercfg-ds4lq\""
Oct 30 19:02:52.945535 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.938333    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5988ce-d59e-4cb6-a637-997e33ba4541-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2\" (UID: \"8b5988ce-d59e-4cb6-a637-997e33ba4541\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:02:52.976631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.973283    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfc6\" (UniqueName: \"kubernetes.io/projected/8b5988ce-d59e-4cb6-a637-997e33ba4541-kube-api-access-cpfc6\") pod \"payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2\" (UID: \"8b5988ce-d59e-4cb6-a637-997e33ba4541\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:02:52.989104 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.987001    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"]
Oct 30 19:02:53.003520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.994904    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"]
Oct 30 19:02:53.003520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:52.995044    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:02:53.050380 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.050351    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:02:53.078488 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.078438    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs59z\" (UniqueName: \"kubernetes.io/projected/6e26ef57-61a1-4045-b15d-a692c4418de3-kube-api-access-hs59z\") pod \"payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l\" (UID: \"6e26ef57-61a1-4045-b15d-a692c4418de3\") " pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:02:53.078920 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.078893    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e26ef57-61a1-4045-b15d-a692c4418de3-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l\" (UID: \"6e26ef57-61a1-4045-b15d-a692c4418de3\") " pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:02:53.185953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.185914    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e26ef57-61a1-4045-b15d-a692c4418de3-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l\" (UID: \"6e26ef57-61a1-4045-b15d-a692c4418de3\") " pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:02:53.186144 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.185989    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs59z\" (UniqueName: \"kubernetes.io/projected/6e26ef57-61a1-4045-b15d-a692c4418de3-kube-api-access-hs59z\") pod \"payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l\" (UID: \"6e26ef57-61a1-4045-b15d-a692c4418de3\") " pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:02:53.187468 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.187440    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e26ef57-61a1-4045-b15d-a692c4418de3-plugins-config-volume\") pod \"payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l\" (UID: \"6e26ef57-61a1-4045-b15d-a692c4418de3\") " pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:02:53.248597 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.248449    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs59z\" (UniqueName: \"kubernetes.io/projected/6e26ef57-61a1-4045-b15d-a692c4418de3-kube-api-access-hs59z\") pod \"payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l\" (UID: \"6e26ef57-61a1-4045-b15d-a692c4418de3\") " pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:02:53.318184 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.318144    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:02:53.479374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.479338    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"]
Oct 30 19:02:53.482658 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:02:53.482620    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5988ce_d59e_4cb6_a637_997e33ba4541.slice/crio-7986590fb9579cba0bd290dd9f2935d0aba5aff07c23248d10e6044d0007713c WatchSource:0}: Error finding container 7986590fb9579cba0bd290dd9f2935d0aba5aff07c23248d10e6044d0007713c: Status 404 returned error can't find the container with id 7986590fb9579cba0bd290dd9f2935d0aba5aff07c23248d10e6044d0007713c
Oct 30 19:02:53.698099 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:02:53.698067    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e26ef57_61a1_4045_b15d_a692c4418de3.slice/crio-8b351c1326ebdd63180520465d35722775ec9782c6cdef452cdfe1655e679c1e WatchSource:0}: Error finding container 8b351c1326ebdd63180520465d35722775ec9782c6cdef452cdfe1655e679c1e: Status 404 returned error can't find the container with id 8b351c1326ebdd63180520465d35722775ec9782c6cdef452cdfe1655e679c1e
Oct 30 19:02:53.699094 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.699073    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"]
Oct 30 19:02:53.707702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.707675    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2" event={"ID":"8b5988ce-d59e-4cb6-a637-997e33ba4541","Type":"ContainerStarted","Data":"af10653dab7844e585396727b188708d5b87df30e338b61311ad031d7b3c3a31"}
Oct 30 19:02:53.707822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.707727    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2" event={"ID":"8b5988ce-d59e-4cb6-a637-997e33ba4541","Type":"ContainerStarted","Data":"7986590fb9579cba0bd290dd9f2935d0aba5aff07c23248d10e6044d0007713c"}
Oct 30 19:02:53.707822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.707766    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:02:53.709229 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.709202    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l" event={"ID":"6e26ef57-61a1-4045-b15d-a692c4418de3","Type":"ContainerStarted","Data":"8b351c1326ebdd63180520465d35722775ec9782c6cdef452cdfe1655e679c1e"}
Oct 30 19:02:53.824822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:53.824755    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2" podStartSLOduration=1.824734987 podStartE2EDuration="1.824734987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:53.819087038 +0000 UTC m=+1464.707004128" watchObservedRunningTime="2026-10-30 19:02:53.824734987 +0000 UTC m=+1464.712652076"
Oct 30 19:02:54.721478 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:54.721427    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l" event={"ID":"6e26ef57-61a1-4045-b15d-a692c4418de3","Type":"ContainerStarted","Data":"7e54e9bb864c93b844a735bd74158a323e5bc9a44347906bb626a562a80635fb"}
Oct 30 19:02:54.721700 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:54.721576    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:02:54.835821 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:54.835763    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l" podStartSLOduration=2.835746784 podStartE2EDuration="2.835746784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:54.833390465 +0000 UTC m=+1465.721307554" watchObservedRunningTime="2026-10-30 19:02:54.835746784 +0000 UTC m=+1465.723663873"
Oct 30 19:02:55.597777 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:55.597743    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:02:55.597963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:55.597838    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:02:56.148384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:56.148337    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"]
Oct 30 19:02:56.241261 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:56.241224    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32f757f5-sw665"]
Oct 30 19:02:57.312518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:57.305443    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f739ba-8f4a-4076-a7eb-17d5d12323b6" path="/var/lib/kubelet/pods/90f739ba-8f4a-4076-a7eb-17d5d12323b6/volumes"
Oct 30 19:02:57.519668 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:57.518583    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:57.625606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:57.622442    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:57.732632 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:57.732581    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:57.733288 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:57.733257    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:02:57.733746 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:57.733724    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": context deadline exceeded"
Oct 30 19:02:57.795653 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:57.795600    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:57.818188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:57.817594    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:57.904481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:57.903598    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.106618 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.106572    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.140636 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.140583    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.273518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.268584    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.416619 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.415574    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.528585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.525632    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.528585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.526307    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:02:58.599106 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.598625    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.637518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.633639    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:02:58.638404 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.638374    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:02:58.817867 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.817765    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.966516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.949207    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:02:58.994517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.971513    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:02:58.994517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.971907    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:58.994517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.994408    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:02:58.994828 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:58.994537    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:02:59.087579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.067069    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:02:59.105517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.093318    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:02:59.127547 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.116134    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.127547 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.122541    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:02:59.140533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.131893    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.140533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.132470    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:02:59.202519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.193638    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.245263 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.245227    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:02:59.326884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.326825    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.336149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.336099    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.528515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.527154    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.567517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.543825    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.642533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.641791    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:59.642533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.642131    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 19:02:59.692514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.653647    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Oct 30 19:02:59.731666 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.731620    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:02:59.764635 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:02:59.764583    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.067537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.043820    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.067537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.044425    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:03:00.067537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.045048    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:03:00.104563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.088714    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.158947 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.158901    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:03:00.351624 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.222237    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.463546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.324020    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.495517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.492581    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:03:00.530520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.511721    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:03:00.675455 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.529717    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.679580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.620720    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:03:00.679777 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.674886    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.724517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.674945    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:00.841516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.806634    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:00.841516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:00.806701    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.045748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.045592    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.104515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.096425    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.104515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.096785    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.104515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.097008    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:03:01.104515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.097187    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:03:01.104515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.097370    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:01.104515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.097404    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.155517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.155403    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:01.155517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.155477    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.235514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.233929    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:01.235514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.233992    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.252600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.252551    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:03:01.405135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.405001    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:01.441486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.441433    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.471679 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.471621    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:01.554519 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:03:01.537343    4267 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.292s"
Oct 30 19:03:01.575567 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.552842    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:03:01.653787 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.653750    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:03:01.732582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:01.715739    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:03:02.020521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.020398    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:02.021312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.020922    4267 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:02.187912 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.173605    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:02.302570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.292166    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 19:03:02.304258 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.304217    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:03:02.305555 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.305528    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:03:02.307387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.307363    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 19:03:02.524575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.499832    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:02.717678 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.717636    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:02.856519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.856103    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:02.875774 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:03:02.875732    4267 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.222s"
Oct 30 19:03:02.876306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.876195    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:03:02.877042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.877028    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:03:02.881423 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.881407    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:03:02.881758 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.881742    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:03:02.923447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.921830    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:03:02.926388 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.926347    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:03:02.927232 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.927129    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:03:02.944481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.942681    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:03:02.961089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.961061    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:03:02.968823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.968696    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:03:02.971012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.970810    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb"
Oct 30 19:03:02.975543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.975521    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:03:02.983619 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:02.983596    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:03:03.006319 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:03.006286    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:03:04.725676 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:04.725640    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:03:05.787673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:05.787644    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:03:19.398172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.398133    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"]
Oct 30 19:03:19.402614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.402590    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:19.459116 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.459081    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"]
Oct 30 19:03:19.465855 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.465822    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dddb3eca-z465p\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:19.468580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.468550    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqgxh\" (UniqueName: \"kubernetes.io/projected/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-kube-api-access-lqgxh\") pod \"maas-api-revoke-sub-dddb3eca-z465p\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:19.468980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.468952    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dddb3eca-z465p\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:19.502359 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.502319    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:19.502582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.502440    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:03:19.570380 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.570348    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dddb3eca-z465p\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:19.570603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.570396    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dddb3eca-z465p\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:19.570603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.570541    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgxh\" (UniqueName: \"kubernetes.io/projected/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-kube-api-access-lqgxh\") pod \"maas-api-revoke-sub-dddb3eca-z465p\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:19.571215 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.571188    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dddb3eca-z465p\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:19.572871 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.572848    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dddb3eca-z465p\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:19.628397 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.628364    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgxh\" (UniqueName: \"kubernetes.io/projected/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-kube-api-access-lqgxh\") pod \"maas-api-revoke-sub-dddb3eca-z465p\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:19.718250 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:19.718208    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:20.101915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:20.101877    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"]
Oct 30 19:03:20.102816 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:03:20.102782    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c4e4c5_6ecd_4354_b636_e818b4c0e793.slice/crio-4b4f45535c59d8f1abc59e28707897039855f7c213103176214314c608e3e3f6 WatchSource:0}: Error finding container 4b4f45535c59d8f1abc59e28707897039855f7c213103176214314c608e3e3f6: Status 404 returned error can't find the container with id 4b4f45535c59d8f1abc59e28707897039855f7c213103176214314c608e3e3f6
Oct 30 19:03:20.161676 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:03:20.161571    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:03:21.110538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.110481    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p" event={"ID":"e3c4e4c5-6ecd-4354-b636-e818b4c0e793","Type":"ContainerStarted","Data":"5d433f909dc6d8fbb94b51f1b2601c27257017c56090497f67240a36822ca5df"}
Oct 30 19:03:21.110957 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.110568    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p" event={"ID":"e3c4e4c5-6ecd-4354-b636-e818b4c0e793","Type":"ContainerStarted","Data":"4b4f45535c59d8f1abc59e28707897039855f7c213103176214314c608e3e3f6"}
Oct 30 19:03:21.223671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.223639    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"]
Oct 30 19:03:21.228222 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.228198    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.230863 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.230807    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p" podStartSLOduration=2.230790706 podStartE2EDuration="2.230790706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:21.224426418 +0000 UTC m=+1492.112343509" watchObservedRunningTime="2026-10-30 19:03:21.230790706 +0000 UTC m=+1492.118707795"
Oct 30 19:03:21.298957 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.298916    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.299199 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.299102    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.299199 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.299188    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gttvz\" (UniqueName: \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-kube-api-access-gttvz\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.299337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.299250    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.299337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.299276    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006a5157-fb29-4104-84a6-c9005120d2e6-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.299337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.299336    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.299539 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.299474    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.299603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.299538    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006a5157-fb29-4104-84a6-c9005120d2e6-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.299785 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.299758    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.317609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.317544    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"]
Oct 30 19:03:21.400953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.400858    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.401123 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.400978    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.401123 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.401018    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gttvz\" (UniqueName: \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-kube-api-access-gttvz\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.401123 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.401067    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.401123 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.401091    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006a5157-fb29-4104-84a6-c9005120d2e6-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.401337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.401126    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.401337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.401218    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.401337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.401252    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006a5157-fb29-4104-84a6-c9005120d2e6-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.401337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.401284    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-credential-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.401337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.401300    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.402104 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.402082    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-certs\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.402460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.402433    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-socket\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.402913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.402887    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-data\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.403322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.403293    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006a5157-fb29-4104-84a6-c9005120d2e6-istiod-ca-cert\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.404337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.404158    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-envoy\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.405833 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.405811    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006a5157-fb29-4104-84a6-c9005120d2e6-istio-podinfo\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.458112 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.458074    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-istio-token\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.458382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.458357    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gttvz\" (UniqueName: \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-kube-api-access-gttvz\") pod \"e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") " pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.557269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.557233    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:21.991313 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:21.991278    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"]
Oct 30 19:03:22.027142 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:03:22.027103    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006a5157_fb29_4104_84a6_c9005120d2e6.slice/crio-35465c69edc103092c7c097b65db24d2f820918fdc3869eefcaf47dc1f8edd86 WatchSource:0}: Error finding container 35465c69edc103092c7c097b65db24d2f820918fdc3869eefcaf47dc1f8edd86: Status 404 returned error can't find the container with id 35465c69edc103092c7c097b65db24d2f820918fdc3869eefcaf47dc1f8edd86
Oct 30 19:03:22.030196 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:22.030154    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:03:22.030331 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:22.030247    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:03:22.030331 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:22.030302    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:03:22.124243 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:22.124210    4267 generic.go:356] "Generic (PLEG): container finished" podID="e3c4e4c5-6ecd-4354-b636-e818b4c0e793" containerID="5d433f909dc6d8fbb94b51f1b2601c27257017c56090497f67240a36822ca5df" exitCode=0
Oct 30 19:03:22.124720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:22.124291    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p" event={"ID":"e3c4e4c5-6ecd-4354-b636-e818b4c0e793","Type":"ContainerDied","Data":"5d433f909dc6d8fbb94b51f1b2601c27257017c56090497f67240a36822ca5df"}
Oct 30 19:03:22.126008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:22.125947    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" event={"ID":"006a5157-fb29-4104-84a6-c9005120d2e6","Type":"ContainerStarted","Data":"35465c69edc103092c7c097b65db24d2f820918fdc3869eefcaf47dc1f8edd86"}
Oct 30 19:03:23.152747 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.152708    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" event={"ID":"006a5157-fb29-4104-84a6-c9005120d2e6","Type":"ContainerStarted","Data":"b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e"}
Oct 30 19:03:23.338520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.338182    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" podStartSLOduration=2.338160776 podStartE2EDuration="2.338160776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:23.337013382 +0000 UTC m=+1494.224930472" watchObservedRunningTime="2026-10-30 19:03:23.338160776 +0000 UTC m=+1494.226077867"
Oct 30 19:03:23.388784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.388757    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:23.559979 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.559941    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 19:03:23.560245 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.560027    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 19:03:23.560612 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.560592    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:23.566329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.566303    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-kube-api-access-lqgxh\" (UniqueName: \"kubernetes.io/projected/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-kube-api-access-lqgxh\") pod \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") "
Oct 30 19:03:23.566425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.566392    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-xks-ca\") pod \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") "
Oct 30 19:03:23.566636 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.566617    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-openshift-ca\") pod \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\" (UID: \"e3c4e4c5-6ecd-4354-b636-e818b4c0e793\") "
Oct 30 19:03:23.568105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.568076    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-openshift-ca" pod "e3c4e4c5-6ecd-4354-b636-e818b4c0e793" (UID: "e3c4e4c5-6ecd-4354-b636-e818b4c0e793"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:23.569835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.569807    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-kube-api-access-lqgxh" pod "e3c4e4c5-6ecd-4354-b636-e818b4c0e793" (UID: "e3c4e4c5-6ecd-4354-b636-e818b4c0e793"). InnerVolumeSpecName "kube-api-access-lqgxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:23.571932 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.571906    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-xks-ca" pod "e3c4e4c5-6ecd-4354-b636-e818b4c0e793" (UID: "e3c4e4c5-6ecd-4354-b636-e818b4c0e793"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:23.668797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.668763    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:23.668797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.668794    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqgxh\" (UniqueName: \"kubernetes.io/projected/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-kube-api-access-lqgxh\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:23.668797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:23.668805    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c4e4c5-6ecd-4354-b636-e818b4c0e793-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:24.163089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:24.163057    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"
Oct 30 19:03:24.163614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:24.163101    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p" event={"ID":"e3c4e4c5-6ecd-4354-b636-e818b4c0e793","Type":"ContainerDied","Data":"4b4f45535c59d8f1abc59e28707897039855f7c213103176214314c608e3e3f6"}
Oct 30 19:03:24.163614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:24.163168    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b4f45535c59d8f1abc59e28707897039855f7c213103176214314c608e3e3f6"
Oct 30 19:03:24.557906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:24.557865    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 19:03:24.558109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:24.557933    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 19:03:25.558145 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:25.558110    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 19:03:25.558581 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:25.558168    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 19:03:26.558586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:26.558546    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 19:03:26.559045 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:26.558618    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 19:03:27.558008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:27.557967    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 19:03:27.558224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:27.558051    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 19:03:28.557916 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:28.557879    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 19:03:28.558316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:28.557944    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 19:03:29.523514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:29.501644    4267 scope.go:122] "RemoveContainer" containerID="8d3cd8c5dda41de1981b1568004866de65f3c4c056e6f4051339499c7cab27bc"
Oct 30 19:03:29.559342 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:29.559305    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 19:03:29.559940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:29.559905    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 19:03:29.632468 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:29.632438    4267 scope.go:122] "RemoveContainer" containerID="6949803b214c0ef6164a235516e025d3c47e2795e7426a7ce56891a8505dc18d"
Oct 30 19:03:30.335514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.308099    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"]
Oct 30 19:03:30.335514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.309201    4267 cpu_manager.go:414] "containerMap: removing container" podUID="e3c4e4c5-6ecd-4354-b636-e818b4c0e793" containerName="revoke-keys"
Oct 30 19:03:30.335514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.309224    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c4e4c5-6ecd-4354-b636-e818b4c0e793" containerName="revoke-keys"
Oct 30 19:03:30.335514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.309407    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c4e4c5-6ecd-4354-b636-e818b4c0e793" containerName="revoke-keys"
Oct 30 19:03:30.383538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.368123    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.448228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.446752    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:03:30.448228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.446855    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:30.476329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.475054    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d55ee1d-wwpcb\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.476329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.475189    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b27f8\" (UniqueName: \"kubernetes.io/projected/259647e9-76ef-4b48-ad76-8d463577981d-kube-api-access-b27f8\") pod \"maas-api-revoke-sub-8d55ee1d-wwpcb\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.476329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.475364    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d55ee1d-wwpcb\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.522594 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.522556    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"]
Oct 30 19:03:30.540570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.534934    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:30.684655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.684621    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-cd57e7dd-serving-cert\""
Oct 30 19:03:30.745756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.740481    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b27f8\" (UniqueName: \"kubernetes.io/projected/259647e9-76ef-4b48-ad76-8d463577981d-kube-api-access-b27f8\") pod \"maas-api-revoke-sub-8d55ee1d-wwpcb\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.745756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.740641    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wwj\" (UniqueName: \"kubernetes.io/projected/72cc29af-2da5-4227-a96c-42482a1e1f00-kube-api-access-m4wwj\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:30.745756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.740773    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d55ee1d-wwpcb\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.745756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.740804    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-maas-api-tls\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:30.745756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.740834    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-metrics-certs\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:30.745756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.740902    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cc29af-2da5-4227-a96c-42482a1e1f00-openshift-service-ca\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:30.745756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.740994    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d55ee1d-wwpcb\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.745756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.741942    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d55ee1d-wwpcb\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.752097 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.752062    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d55ee1d-wwpcb\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.773471 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.773435    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"]
Oct 30 19:03:30.804703 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.804670    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b27f8\" (UniqueName: \"kubernetes.io/projected/259647e9-76ef-4b48-ad76-8d463577981d-kube-api-access-b27f8\") pod \"maas-api-revoke-sub-8d55ee1d-wwpcb\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.805217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.805193    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:30.806720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.806701    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:30.827507 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.809037    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:03:30.827507 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.821005    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:30.859714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.859439    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wwj\" (UniqueName: \"kubernetes.io/projected/72cc29af-2da5-4227-a96c-42482a1e1f00-kube-api-access-m4wwj\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:30.859714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.859598    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-maas-api-tls\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:30.859714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.859630    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-metrics-certs\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:30.863919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.860836    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cc29af-2da5-4227-a96c-42482a1e1f00-openshift-service-ca\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:30.863919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.862146    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cc29af-2da5-4227-a96c-42482a1e1f00-openshift-service-ca\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:30.863919 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:03:30.863694    4267 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-serving-cert: secret "maas-api-e2e-derive-cd57e7dd-serving-cert" not found
Oct 30 19:03:30.863919 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:03:30.863778    4267 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-maas-api-tls podName:72cc29af-2da5-4227-a96c-42482a1e1f00 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:31.363756929 +0000 UTC m=+1502.251674010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-maas-api-tls") pod "maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl" (UID: "72cc29af-2da5-4227-a96c-42482a1e1f00") : secret "maas-api-e2e-derive-cd57e7dd-serving-cert" not found
Oct 30 19:03:30.923148 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:30.922816    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-metrics-certs\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:31.005541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:31.002551    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wwj\" (UniqueName: \"kubernetes.io/projected/72cc29af-2da5-4227-a96c-42482a1e1f00-kube-api-access-m4wwj\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:31.005541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:31.002629    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"]
Oct 30 19:03:31.455195 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:31.455159    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-maas-api-tls\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:31.457924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:31.457892    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-maas-api-tls\") pod \"maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:31.594361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:31.594324    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"]
Oct 30 19:03:31.633616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:31.633587    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:32.075351 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:03:32.075320    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cc29af_2da5_4227_a96c_42482a1e1f00.slice/crio-2b8aaa6a209fbe0c5e40e99267ae42d6f7373b00891c2cd7853a57c6fa8cc8ca WatchSource:0}: Error finding container 2b8aaa6a209fbe0c5e40e99267ae42d6f7373b00891c2cd7853a57c6fa8cc8ca: Status 404 returned error can't find the container with id 2b8aaa6a209fbe0c5e40e99267ae42d6f7373b00891c2cd7853a57c6fa8cc8ca
Oct 30 19:03:32.076797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:32.076771    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"]
Oct 30 19:03:32.287690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:32.287652    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb" event={"ID":"259647e9-76ef-4b48-ad76-8d463577981d","Type":"ContainerStarted","Data":"fd32e8e074d56d586965fa96c9b5f74a2099827c6f40729df16eb86ad9eae986"}
Oct 30 19:03:32.287904 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:32.287721    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb" event={"ID":"259647e9-76ef-4b48-ad76-8d463577981d","Type":"ContainerStarted","Data":"ca1f7a68177adfb57b9e9fdc8e0af16cc1406ca39f21c53a4c37e0f99a489163"}
Oct 30 19:03:32.289339 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:32.289312    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl" event={"ID":"72cc29af-2da5-4227-a96c-42482a1e1f00","Type":"ContainerStarted","Data":"2b8aaa6a209fbe0c5e40e99267ae42d6f7373b00891c2cd7853a57c6fa8cc8ca"}
Oct 30 19:03:32.401166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:32.401101    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb" podStartSLOduration=2.401085115 podStartE2EDuration="2.401085115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:32.399028316 +0000 UTC m=+1503.286945429" watchObservedRunningTime="2026-10-30 19:03:32.401085115 +0000 UTC m=+1503.289002203"
Oct 30 19:03:33.300181 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:33.300144    4267 generic.go:356] "Generic (PLEG): container finished" podID="259647e9-76ef-4b48-ad76-8d463577981d" containerID="fd32e8e074d56d586965fa96c9b5f74a2099827c6f40729df16eb86ad9eae986" exitCode=0
Oct 30 19:03:33.300700 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:33.300229    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb" event={"ID":"259647e9-76ef-4b48-ad76-8d463577981d","Type":"ContainerDied","Data":"fd32e8e074d56d586965fa96c9b5f74a2099827c6f40729df16eb86ad9eae986"}
Oct 30 19:03:33.302841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:33.302809    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl" event={"ID":"72cc29af-2da5-4227-a96c-42482a1e1f00","Type":"ContainerStarted","Data":"ec6c0a8b288a8ae86b870fe39f9568e20b3de280d48bc9e8cd27053193f4e729"}
Oct 30 19:03:33.302957 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:33.302882    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:33.533330 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:33.533263    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl" podStartSLOduration=2.679626349 podStartE2EDuration="3.533242656s" totalImagesPullingTime="853.616307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:33.528579797 +0000 UTC m=+1504.416496888" watchObservedRunningTime="2026-10-30 19:03:33.533242656 +0000 UTC m=+1504.421159748"
Oct 30 19:03:34.472409 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:34.472381    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:34.607742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:34.607655    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259647e9-76ef-4b48-ad76-8d463577981d-kube-api-access-b27f8\" (UniqueName: \"kubernetes.io/projected/259647e9-76ef-4b48-ad76-8d463577981d-kube-api-access-b27f8\") pod \"259647e9-76ef-4b48-ad76-8d463577981d\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") "
Oct 30 19:03:34.607903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:34.607798    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-openshift-ca\") pod \"259647e9-76ef-4b48-ad76-8d463577981d\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") "
Oct 30 19:03:34.607903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:34.607840    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-xks-ca\") pod \"259647e9-76ef-4b48-ad76-8d463577981d\" (UID: \"259647e9-76ef-4b48-ad76-8d463577981d\") "
Oct 30 19:03:34.608153 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:34.608130    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-openshift-ca" pod "259647e9-76ef-4b48-ad76-8d463577981d" (UID: "259647e9-76ef-4b48-ad76-8d463577981d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:34.608642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:34.608622    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:34.611933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:34.611905    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259647e9-76ef-4b48-ad76-8d463577981d-kube-api-access-b27f8" pod "259647e9-76ef-4b48-ad76-8d463577981d" (UID: "259647e9-76ef-4b48-ad76-8d463577981d"). InnerVolumeSpecName "kube-api-access-b27f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:34.611933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:34.611911    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-xks-ca" pod "259647e9-76ef-4b48-ad76-8d463577981d" (UID: "259647e9-76ef-4b48-ad76-8d463577981d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:34.710334 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:34.710297    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b27f8\" (UniqueName: \"kubernetes.io/projected/259647e9-76ef-4b48-ad76-8d463577981d-kube-api-access-b27f8\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:34.710334 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:34.710330    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259647e9-76ef-4b48-ad76-8d463577981d-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:03:35.322265 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:35.322216    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb" event={"ID":"259647e9-76ef-4b48-ad76-8d463577981d","Type":"ContainerDied","Data":"ca1f7a68177adfb57b9e9fdc8e0af16cc1406ca39f21c53a4c37e0f99a489163"}
Oct 30 19:03:35.322476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:35.322324    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1f7a68177adfb57b9e9fdc8e0af16cc1406ca39f21c53a4c37e0f99a489163"
Oct 30 19:03:35.322476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:35.322427    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"
Oct 30 19:03:39.317360 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:39.317329    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:03:50.162272 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:03:50.162229    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:03:59.053125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:59.053079    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"]
Oct 30 19:03:59.101641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:59.101606    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed98e73a-zdbj9"]
Oct 30 19:03:59.225759 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:03:59.225717    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6d0fe0-0ad4-4d5a-9f48-68c545841baf" path="/var/lib/kubelet/pods/bd6d0fe0-0ad4-4d5a-9f48-68c545841baf/volumes"
Oct 30 19:04:00.194425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.194390    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"]
Oct 30 19:04:00.195251 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.195228    4267 cpu_manager.go:414] "containerMap: removing container" podUID="259647e9-76ef-4b48-ad76-8d463577981d" containerName="revoke-keys"
Oct 30 19:04:00.195358 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.195253    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259647e9-76ef-4b48-ad76-8d463577981d" containerName="revoke-keys"
Oct 30 19:04:00.195436 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.195421    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="259647e9-76ef-4b48-ad76-8d463577981d" containerName="revoke-keys"
Oct 30 19:04:00.200356 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.200334    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.244501 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.244452    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"]
Oct 30 19:04:00.248984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.248962    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.252169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.252139    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qwc8z\""
Oct 30 19:04:00.283086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.283044    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.283270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.283093    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.283270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.283128    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.283270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.283236    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.283693 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.283599    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.283882 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.283853    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgb7l\" (UniqueName: \"kubernetes.io/projected/38d93a40-3273-466a-9475-fa0960ed3e35-kube-api-access-qgb7l\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.284004 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.283981    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d93a40-3273-466a-9475-fa0960ed3e35-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.284062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284017    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.284107 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284087    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.284182 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284166    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.284243 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284229    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d93a40-3273-466a-9475-fa0960ed3e35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.284281 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284270    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d93a40-3273-466a-9475-fa0960ed3e35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.284338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284323    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.284416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284402    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.284455 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284428    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2lzc\" (UniqueName: \"kubernetes.io/projected/96d47ea1-b235-45d8-8573-0dae95cd754d-kube-api-access-t2lzc\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.284541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284525    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d47ea1-b235-45d8-8573-0dae95cd754d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.284587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284559    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.284627 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.284614    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.351382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.351333    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"]
Oct 30 19:04:00.386247 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386211    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d93a40-3273-466a-9475-fa0960ed3e35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.386447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386273    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.386447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386337    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.386447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386362    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2lzc\" (UniqueName: \"kubernetes.io/projected/96d47ea1-b235-45d8-8573-0dae95cd754d-kube-api-access-t2lzc\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.386447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386427    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d47ea1-b235-45d8-8573-0dae95cd754d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.386728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386452    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.386728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386504    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.386728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386549    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.386728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386575    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.386728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386609    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.386728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386654    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.386728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386714    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.387091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386737    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgb7l\" (UniqueName: \"kubernetes.io/projected/38d93a40-3273-466a-9475-fa0960ed3e35-kube-api-access-qgb7l\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.387091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386829    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.387091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386832    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d93a40-3273-466a-9475-fa0960ed3e35-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.387091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386900    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.387091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386986    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.387091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.387059    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.387091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.387069    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.387538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.387161    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d93a40-3273-466a-9475-fa0960ed3e35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.387538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.387354    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.387538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.387379    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.387538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.386829    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.387970 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.387940    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d93a40-3273-466a-9475-fa0960ed3e35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.388091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.388039    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.388708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.388677    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.388889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.388862    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d47ea1-b235-45d8-8573-0dae95cd754d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.389207 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.389175    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.390076 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.390033    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.390349 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.390321    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d93a40-3273-466a-9475-fa0960ed3e35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.391158 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.391138    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d93a40-3273-466a-9475-fa0960ed3e35-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.391709 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.391687    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.405415 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.405383    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"]
Oct 30 19:04:00.444609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.444526    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2lzc\" (UniqueName: \"kubernetes.io/projected/96d47ea1-b235-45d8-8573-0dae95cd754d-kube-api-access-t2lzc\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.447301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.447243    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d93a40-3273-466a-9475-fa0960ed3e35-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.450098 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.450066    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgb7l\" (UniqueName: \"kubernetes.io/projected/38d93a40-3273-466a-9475-fa0960ed3e35-kube-api-access-qgb7l\") pod \"maas-default-gateway-openshift-default-6db586dd6c-x8kbj\" (UID: \"38d93a40-3273-466a-9475-fa0960ed3e35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.453822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.453771    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d47ea1-b235-45d8-8573-0dae95cd754d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-864kk\" (UID: \"96d47ea1-b235-45d8-8573-0dae95cd754d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.518585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.518546    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:00.565542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.565512    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:00.961542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.961474    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"]
Oct 30 19:04:00.996126 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:04:00.996089    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d47ea1_b235_45d8_8573_0dae95cd754d.slice/crio-15c65d951f7bcc2244d704cc884fcccc5a00e497353f2c8769d56f26edf81e3b WatchSource:0}: Error finding container 15c65d951f7bcc2244d704cc884fcccc5a00e497353f2c8769d56f26edf81e3b: Status 404 returned error can't find the container with id 15c65d951f7bcc2244d704cc884fcccc5a00e497353f2c8769d56f26edf81e3b
Oct 30 19:04:00.998624 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.998586    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:00.998710 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.998684    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:00.998748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:00.998717    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:01.012905 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:04:01.012872    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d93a40_3273_466a_9475_fa0960ed3e35.slice/crio-7d5ed2ddeabd33a96cba5282035a91cc445ad182bdba095113fdf27a052167ed WatchSource:0}: Error finding container 7d5ed2ddeabd33a96cba5282035a91cc445ad182bdba095113fdf27a052167ed: Status 404 returned error can't find the container with id 7d5ed2ddeabd33a96cba5282035a91cc445ad182bdba095113fdf27a052167ed
Oct 30 19:04:01.014014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:01.013988    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"]
Oct 30 19:04:01.016302 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:01.016264    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:01.016381 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:01.016358    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:01.016429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:01.016410    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:01.582749 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:01.582713    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" event={"ID":"38d93a40-3273-466a-9475-fa0960ed3e35","Type":"ContainerStarted","Data":"0dc61a2193eed9fb41cc44829d750243c690630b10532969e71fbefb1b5a4f26"}
Oct 30 19:04:01.583223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:01.582778    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" event={"ID":"38d93a40-3273-466a-9475-fa0960ed3e35","Type":"ContainerStarted","Data":"7d5ed2ddeabd33a96cba5282035a91cc445ad182bdba095113fdf27a052167ed"}
Oct 30 19:04:01.584722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:01.584693    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" event={"ID":"96d47ea1-b235-45d8-8573-0dae95cd754d","Type":"ContainerStarted","Data":"83a6b176a03f87ac14a9871043fdbc54b2713bc7628914526d7657d2dd34c77e"}
Oct 30 19:04:01.584869 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:01.584748    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" event={"ID":"96d47ea1-b235-45d8-8573-0dae95cd754d","Type":"ContainerStarted","Data":"15c65d951f7bcc2244d704cc884fcccc5a00e497353f2c8769d56f26edf81e3b"}
Oct 30 19:04:01.706975 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:01.706918    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" podStartSLOduration=1.706902559 podStartE2EDuration="1.706902559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:01.702281127 +0000 UTC m=+1532.590198217" watchObservedRunningTime="2026-10-30 19:04:01.706902559 +0000 UTC m=+1532.594819648"
Oct 30 19:04:02.542514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:02.530418    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:02.542514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:02.531248    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-864kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:02.542514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:02.531296    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" podUID="96d47ea1-b235-45d8-8573-0dae95cd754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 19:04:02.570516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:02.569722    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:02.570516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:02.569904    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8kbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:02.570516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:02.569951    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" podUID="38d93a40-3273-466a-9475-fa0960ed3e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 19:04:03.189549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.187651    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" podStartSLOduration=3.187634785 podStartE2EDuration="3.187634785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:01.83791006 +0000 UTC m=+1532.725827149" watchObservedRunningTime="2026-10-30 19:04:03.187634785 +0000 UTC m=+1534.075551873"
Oct 30 19:04:03.193132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.193101    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"]
Oct 30 19:04:03.198268 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.198240    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.255955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.255923    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-36d676-openshift-default-dockercfg-nc5q4\""
Oct 30 19:04:03.625712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.625618    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-864kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:03.625999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.625961    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" podUID="96d47ea1-b235-45d8-8573-0dae95cd754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 19:04:03.648819 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.628301    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2kkh\" (UniqueName: \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-kube-api-access-m2kkh\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.655066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.653616    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6af68d-aabc-4ced-b194-395f48c4e36c-istiod-ca-cert\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.655066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.653747    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-token\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.655066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.653788    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-podinfo\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.655066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.653821    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-data\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.655066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.653942    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-certs\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.655066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.653976    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-envoy\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.655066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.654216    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-credential-socket\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.655066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.654532    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-socket\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.761183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.734187    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8kbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:03.761183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.734252    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" podUID="38d93a40-3273-466a-9475-fa0960ed3e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 19:04:03.781592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.781548    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-socket\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.815812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.785327    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-socket\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.816011 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.785403    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"]
Oct 30 19:04:03.816219 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.816191    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2kkh\" (UniqueName: \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-kube-api-access-m2kkh\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.816372 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.816358    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6af68d-aabc-4ced-b194-395f48c4e36c-istiod-ca-cert\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.816551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.816514    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-token\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.820606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.820572    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-podinfo\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.820788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.820772    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-data\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.820959 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.820944    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-certs\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.840620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.840581    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-envoy\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.841042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.841011    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-credential-socket\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.841743 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.841720    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-credential-socket\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.842227 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.842194    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-data\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.871149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.829659    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6af68d-aabc-4ced-b194-395f48c4e36c-istiod-ca-cert\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.871627 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.862134    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-certs\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.914866 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.883734    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-podinfo\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.914866 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.886787    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-envoy\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.920206 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.920168    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-token\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:03.922617 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:03.921018    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2kkh\" (UniqueName: \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-kube-api-access-m2kkh\") pod \"e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") " pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:04.163524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:04.157817    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:04.579655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:04.574820    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-864kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:04.579655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:04.574890    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" podUID="96d47ea1-b235-45d8-8573-0dae95cd754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 19:04:04.581828 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:04.581473    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8kbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:04.581828 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:04.581547    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" podUID="38d93a40-3273-466a-9475-fa0960ed3e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 19:04:04.877337 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:04:04.876756    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6af68d_aabc_4ced_b194_395f48c4e36c.slice/crio-43d898e7193919a5562798750e89550795513b91238b60cb9e6120031523e9ce WatchSource:0}: Error finding container 43d898e7193919a5562798750e89550795513b91238b60cb9e6120031523e9ce: Status 404 returned error can't find the container with id 43d898e7193919a5562798750e89550795513b91238b60cb9e6120031523e9ce
Oct 30 19:04:04.879784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:04.879742    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:04.879983 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:04.879964    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:04.880109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:04.880094    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:04.884080 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:04.883928    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"]
Oct 30 19:04:05.540512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:05.539600    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-864kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:05.540512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:05.539663    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" podUID="96d47ea1-b235-45d8-8573-0dae95cd754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 19:04:05.575000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:05.574963    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8kbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:05.585576 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:05.575224    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" podUID="38d93a40-3273-466a-9475-fa0960ed3e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 19:04:05.795036 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:05.794929    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" event={"ID":"5f6af68d-aabc-4ced-b194-395f48c4e36c","Type":"ContainerStarted","Data":"fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f"}
Oct 30 19:04:05.795036 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:05.794996    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" event={"ID":"5f6af68d-aabc-4ced-b194-395f48c4e36c","Type":"ContainerStarted","Data":"43d898e7193919a5562798750e89550795513b91238b60cb9e6120031523e9ce"}
Oct 30 19:04:05.984453 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:05.984319    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" podStartSLOduration=2.984296741 podStartE2EDuration="2.984296741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:05.932879403 +0000 UTC m=+1536.820796494" watchObservedRunningTime="2026-10-30 19:04:05.984296741 +0000 UTC m=+1536.872213830"
Oct 30 19:04:06.184625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:06.179780    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:06.208523 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:06.196859    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:06.521172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:06.521062    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-864kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:06.521473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:06.521424    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" podUID="96d47ea1-b235-45d8-8573-0dae95cd754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 19:04:06.567537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:06.567008    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8kbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:06.567537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:06.567073    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" podUID="38d93a40-3273-466a-9475-fa0960ed3e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 19:04:06.803093 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:06.803009    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:06.803987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:06.803967    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:04:07.550518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:07.539618    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-864kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:07.550518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:07.539686    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" podUID="96d47ea1-b235-45d8-8573-0dae95cd754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 19:04:07.566548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:07.566192    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8kbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:07.566548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:07.566260    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" podUID="38d93a40-3273-466a-9475-fa0960ed3e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 19:04:08.519768 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:08.519725    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-864kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:04:08.520182 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:08.519798    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" podUID="96d47ea1-b235-45d8-8573-0dae95cd754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 19:04:08.565999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:08.565952    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8kbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:04:08.566155 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:08.566023    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" podUID="38d93a40-3273-466a-9475-fa0960ed3e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 19:04:10.524520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:10.523820    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-864kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:04:10.524520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:10.523893    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" podUID="96d47ea1-b235-45d8-8573-0dae95cd754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:10.543012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:10.542962    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:10.543955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:10.543933    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:10.544668 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:10.544648    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk"
Oct 30 19:04:10.553259 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:10.553221    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:10.554796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:10.554773    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:10.555742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:10.555722    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj"
Oct 30 19:04:13.563516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.552629    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"]
Oct 30 19:04:13.605518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.604582    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"]
Oct 30 19:04:13.605518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.604716    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:04:13.711349 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.711312    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-36d676\""
Oct 30 19:04:13.711349 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.711317    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-36d676-dockercfg-mgbt4\""
Oct 30 19:04:13.792327 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.792282    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62062c97-8d55-4933-b39e-8961ed4925b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj\" (UID: \"62062c97-8d55-4933-b39e-8961ed4925b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:04:13.792557 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.792486    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnqb9\" (UniqueName: \"kubernetes.io/projected/62062c97-8d55-4933-b39e-8961ed4925b9-kube-api-access-wnqb9\") pod \"payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj\" (UID: \"62062c97-8d55-4933-b39e-8961ed4925b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:04:13.894089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.893999    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnqb9\" (UniqueName: \"kubernetes.io/projected/62062c97-8d55-4933-b39e-8961ed4925b9-kube-api-access-wnqb9\") pod \"payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj\" (UID: \"62062c97-8d55-4933-b39e-8961ed4925b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:04:13.894259 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.894197    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62062c97-8d55-4933-b39e-8961ed4925b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj\" (UID: \"62062c97-8d55-4933-b39e-8961ed4925b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:04:13.895037 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.895008    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62062c97-8d55-4933-b39e-8961ed4925b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj\" (UID: \"62062c97-8d55-4933-b39e-8961ed4925b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:04:13.953937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.953901    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnqb9\" (UniqueName: \"kubernetes.io/projected/62062c97-8d55-4933-b39e-8961ed4925b9-kube-api-access-wnqb9\") pod \"payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj\" (UID: \"62062c97-8d55-4933-b39e-8961ed4925b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:04:13.963330 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:13.963296    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:04:14.354038 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:04:14.354000    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62062c97_8d55_4933_b39e_8961ed4925b9.slice/crio-ddde7acecb9735021410ae5d91360f533c33b2537fe903a077a66fe12f681e32 WatchSource:0}: Error finding container ddde7acecb9735021410ae5d91360f533c33b2537fe903a077a66fe12f681e32: Status 404 returned error can't find the container with id ddde7acecb9735021410ae5d91360f533c33b2537fe903a077a66fe12f681e32
Oct 30 19:04:14.355442 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:14.355413    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"]
Oct 30 19:04:14.890154 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:14.890115    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj" event={"ID":"62062c97-8d55-4933-b39e-8961ed4925b9","Type":"ContainerStarted","Data":"edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641"}
Oct 30 19:04:14.890594 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:14.890181    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj" event={"ID":"62062c97-8d55-4933-b39e-8961ed4925b9","Type":"ContainerStarted","Data":"ddde7acecb9735021410ae5d91360f533c33b2537fe903a077a66fe12f681e32"}
Oct 30 19:04:14.890594 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:14.890208    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:04:15.008307 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:15.008253    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj" podStartSLOduration=2.008234881 podStartE2EDuration="2.008234881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:15.005097548 +0000 UTC m=+1545.893014636" watchObservedRunningTime="2026-10-30 19:04:15.008234881 +0000 UTC m=+1545.896151970"
Oct 30 19:04:15.061132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:15.061095    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"]
Oct 30 19:04:15.065963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:15.065937    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ac7525-8gz9z"]
Oct 30 19:04:15.225253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:15.225221    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8" path="/var/lib/kubelet/pods/1fe7c242-5a25-4a5e-be24-ce5c6b7d08e8/volumes"
Oct 30 19:04:20.161311 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:04:20.161269    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:04:22.445527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.444434    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"]
Oct 30 19:04:22.502524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.501954    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.531589 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.530027    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"]
Oct 30 19:04:22.544668 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.544627    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-certs\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.544820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.544703    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f3d4ae-3622-413b-9038-debcc28bec24-istiod-ca-cert\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.544820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.544769    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-data\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.544952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.544871    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmvz\" (UniqueName: \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-kube-api-access-wdmvz\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.544952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.544928    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-istio-token\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.545054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.544968    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-credential-socket\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.545054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.545034    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-socket\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.545162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.545073    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-envoy\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.545217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.545186    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f3d4ae-3622-413b-9038-debcc28bec24-istio-podinfo\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.586924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.586888    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-7d059d-openshift-default-dockercfg-68fm8\""
Oct 30 19:04:22.646432 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.646377    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmvz\" (UniqueName: \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-kube-api-access-wdmvz\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.646671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.646456    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-istio-token\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.647605 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.646514    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-credential-socket\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.648250 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.648222    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-credential-socket\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.648348 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.648298    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-socket\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.648405 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.648373    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-envoy\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.648583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.648564    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f3d4ae-3622-413b-9038-debcc28bec24-istio-podinfo\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.648821 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.648801    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-certs\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.648889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.648869    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f3d4ae-3622-413b-9038-debcc28bec24-istiod-ca-cert\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.648964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.648950    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-data\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.649353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.649331    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-data\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.649413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.648809    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-socket\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.649584 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.649561    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-certs\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.650040 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.650012    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f3d4ae-3622-413b-9038-debcc28bec24-istiod-ca-cert\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.653040 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.653005    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-envoy\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.653477 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.653457    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f3d4ae-3622-413b-9038-debcc28bec24-istio-podinfo\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.704660 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.704589    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-istio-token\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.706566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.706441    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmvz\" (UniqueName: \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-kube-api-access-wdmvz\") pod \"e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") " pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:22.847515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:22.847147    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:23.295644 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:04:23.295602    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f3d4ae_3622_413b_9038_debcc28bec24.slice/crio-2e16200266888f511c117e336c7bd0294ae8ea9a3698bde3bd647965eacf963b WatchSource:0}: Error finding container 2e16200266888f511c117e336c7bd0294ae8ea9a3698bde3bd647965eacf963b: Status 404 returned error can't find the container with id 2e16200266888f511c117e336c7bd0294ae8ea9a3698bde3bd647965eacf963b
Oct 30 19:04:23.297595 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:23.297570    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"]
Oct 30 19:04:23.298572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:23.298538    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:23.298676 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:23.298608    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:23.298676 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:23.298641    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:04:23.974868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:23.974827    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" event={"ID":"41f3d4ae-3622-413b-9038-debcc28bec24","Type":"ContainerStarted","Data":"ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44"}
Oct 30 19:04:23.975274 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:23.974891    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" event={"ID":"41f3d4ae-3622-413b-9038-debcc28bec24","Type":"ContainerStarted","Data":"2e16200266888f511c117e336c7bd0294ae8ea9a3698bde3bd647965eacf963b"}
Oct 30 19:04:24.095717 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:24.095665    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" podStartSLOduration=2.095650297 podStartE2EDuration="2.095650297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:24.092994604 +0000 UTC m=+1554.980911696" watchObservedRunningTime="2026-10-30 19:04:24.095650297 +0000 UTC m=+1554.983567386"
Oct 30 19:04:24.848387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:24.848345    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:24.854512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:24.854471    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:24.983562 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:24.983526    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:24.984736 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:24.984714    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:04:25.902184 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:25.902154    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:04:29.718527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:29.718468    4267 scope.go:122] "RemoveContainer" containerID="fcc3bfa7f210cefbc9a78cfef6d7c632ce6da0a4562b6c759e02c43ad60804e7"
Oct 30 19:04:29.785373 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:29.785252    4267 scope.go:122] "RemoveContainer" containerID="398ccdf2c5d0da34bd32cf2dd715d4b22c69b63a8248d986106dda1a5b120c25"
Oct 30 19:04:29.843810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:29.843781    4267 scope.go:122] "RemoveContainer" containerID="41858990bea9c7afde62e92298d002af3397cb117683b88fed01788a566cd3b4"
Oct 30 19:04:30.684886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:30.684830    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:04:30.685224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:30.685178    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:30.897529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:30.895596    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:31.040569 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.039675    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:31.114642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.114599    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:31.194514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.188854    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:31.218522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.217747    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"]
Oct 30 19:04:31.218726 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.218691    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb" podUID="e108a863-9bb1-46e7-a1ca-93c23dc91d92" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:31.276379 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.276347    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:04:31.329854 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.329746    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw\" (UID: \"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:04:31.330549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.330510    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9j9\" (UniqueName: \"kubernetes.io/projected/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-kube-api-access-jz9j9\") pod \"payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw\" (UID: \"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:04:31.333225 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.333204    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-7d059d\""
Oct 30 19:04:31.391018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.390844    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-7d059d-dockercfg-4hgpr\""
Oct 30 19:04:31.456518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.434685    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9j9\" (UniqueName: \"kubernetes.io/projected/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-kube-api-access-jz9j9\") pod \"payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw\" (UID: \"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:04:31.458104 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.458069    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw\" (UID: \"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:04:31.493947 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.459129    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw\" (UID: \"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:04:31.551652 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.551597    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:31.553616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.553585    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9j9\" (UniqueName: \"kubernetes.io/projected/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-kube-api-access-jz9j9\") pod \"payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw\" (UID: \"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:04:31.602056 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.601963    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:04:31.658519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.653058    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded"
Oct 30 19:04:31.668189 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.668154    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"]
Oct 30 19:04:31.734091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:31.734061    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:04:32.161534 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:04:32.161470    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f19a3e_b6d1_4ecb_aba1_5497d09d6bc9.slice/crio-ff8277860ffb24df5f45048444f018d528e65e9864412a1f2357a9b80eb51cc9 WatchSource:0}: Error finding container ff8277860ffb24df5f45048444f018d528e65e9864412a1f2357a9b80eb51cc9: Status 404 returned error can't find the container with id ff8277860ffb24df5f45048444f018d528e65e9864412a1f2357a9b80eb51cc9
Oct 30 19:04:32.162582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:32.162550    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"]
Oct 30 19:04:32.879985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:32.879946    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw" event={"ID":"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9","Type":"ContainerStarted","Data":"f7fac6684d17ea0d4912d64c95d0212279f67938c8ab1a99b69235e10f12d0b6"}
Oct 30 19:04:32.880199 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:32.880007    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:04:32.880199 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:32.880023    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw" event={"ID":"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9","Type":"ContainerStarted","Data":"ff8277860ffb24df5f45048444f018d528e65e9864412a1f2357a9b80eb51cc9"}
Oct 30 19:04:32.994485 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:32.994427    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw" podStartSLOduration=2.99441276 podStartE2EDuration="2.99441276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:32.992019126 +0000 UTC m=+1563.879936215" watchObservedRunningTime="2026-10-30 19:04:32.99441276 +0000 UTC m=+1563.882329903"
Oct 30 19:04:33.640019 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:33.639983    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"]
Oct 30 19:04:33.749246 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:33.749215    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"]
Oct 30 19:04:38.640328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:38.640260    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd" podUID="9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66" containerName="payload-processing" containerID="cri-o://c6006eb49f00b8f069b92f0a77307d639ff7ad1dc16d68738c8cc8d0e7ecbb53" gracePeriod=25
Oct 30 19:04:38.750068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:38.750000    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn" podUID="c6a2d6fd-783b-4e85-b831-0ad295744988" containerName="payload-pre-processing" containerID="cri-o://d027eadc3c7bfbe1d14004061e72e5e80753c2ca290ccb53984e1d829ada50ac" gracePeriod=25
Oct 30 19:04:38.943616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:38.943585    4267 generic.go:356] "Generic (PLEG): container finished" podID="c6a2d6fd-783b-4e85-b831-0ad295744988" containerID="d027eadc3c7bfbe1d14004061e72e5e80753c2ca290ccb53984e1d829ada50ac" exitCode=0
Oct 30 19:04:38.943759 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:38.943659    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn" event={"ID":"c6a2d6fd-783b-4e85-b831-0ad295744988","Type":"ContainerDied","Data":"d027eadc3c7bfbe1d14004061e72e5e80753c2ca290ccb53984e1d829ada50ac"}
Oct 30 19:04:38.945272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:38.945247    4267 generic.go:356] "Generic (PLEG): container finished" podID="9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66" containerID="c6006eb49f00b8f069b92f0a77307d639ff7ad1dc16d68738c8cc8d0e7ecbb53" exitCode=0
Oct 30 19:04:38.945402 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:38.945284    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd" event={"ID":"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66","Type":"ContainerDied","Data":"c6006eb49f00b8f069b92f0a77307d639ff7ad1dc16d68738c8cc8d0e7ecbb53"}
Oct 30 19:04:38.945402 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:38.945315    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd" event={"ID":"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66","Type":"ContainerDied","Data":"6ecb5aac1ce24a434500b523d0df05fb215565bd3925de215ca53c6a793ac168"}
Oct 30 19:04:38.945402 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:38.945331    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ecb5aac1ce24a434500b523d0df05fb215565bd3925de215ca53c6a793ac168"
Oct 30 19:04:38.949345 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:38.949322    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:04:39.037237 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.037211    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-kube-api-access-db8x9\" (UniqueName: \"kubernetes.io/projected/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-kube-api-access-db8x9\") pod \"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66\" (UID: \"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66\") "
Oct 30 19:04:39.037397 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.037376    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-plugins-config-volume\") pod \"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66\" (UID: \"9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66\") "
Oct 30 19:04:39.038040 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.038011    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-plugins-config-volume" pod "9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66" (UID: "9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:39.038464 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.038434    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:39.039712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.039621    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-kube-api-access-db8x9" pod "9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66" (UID: "9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66"). InnerVolumeSpecName "kube-api-access-db8x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:39.058934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.058907    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:04:39.139916 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.139877    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a2d6fd-783b-4e85-b831-0ad295744988-kube-api-access-642lk\" (UniqueName: \"kubernetes.io/projected/c6a2d6fd-783b-4e85-b831-0ad295744988-kube-api-access-642lk\") pod \"c6a2d6fd-783b-4e85-b831-0ad295744988\" (UID: \"c6a2d6fd-783b-4e85-b831-0ad295744988\") "
Oct 30 19:04:39.140275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.140257    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a2d6fd-783b-4e85-b831-0ad295744988-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a2d6fd-783b-4e85-b831-0ad295744988-plugins-config-volume\") pod \"c6a2d6fd-783b-4e85-b831-0ad295744988\" (UID: \"c6a2d6fd-783b-4e85-b831-0ad295744988\") "
Oct 30 19:04:39.140675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.140644    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a2d6fd-783b-4e85-b831-0ad295744988-plugins-config-volume" pod "c6a2d6fd-783b-4e85-b831-0ad295744988" (UID: "c6a2d6fd-783b-4e85-b831-0ad295744988"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:39.140958 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.140942    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a2d6fd-783b-4e85-b831-0ad295744988-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:39.141028 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.140959    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db8x9\" (UniqueName: \"kubernetes.io/projected/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66-kube-api-access-db8x9\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:39.142109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.142089    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a2d6fd-783b-4e85-b831-0ad295744988-kube-api-access-642lk" pod "c6a2d6fd-783b-4e85-b831-0ad295744988" (UID: "c6a2d6fd-783b-4e85-b831-0ad295744988"). InnerVolumeSpecName "kube-api-access-642lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:39.245278 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.245245    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-642lk\" (UniqueName: \"kubernetes.io/projected/c6a2d6fd-783b-4e85-b831-0ad295744988-kube-api-access-642lk\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:39.958177 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.958143    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"
Oct 30 19:04:39.958177 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.958151    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn" event={"ID":"c6a2d6fd-783b-4e85-b831-0ad295744988","Type":"ContainerDied","Data":"5684e4a5f7f5a90d28e0a8f72455b12cbe1001d02900c0dcffc34b98541e978f"}
Oct 30 19:04:39.958764 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.958149    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"
Oct 30 19:04:39.958764 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:39.958267    4267 scope.go:122] "RemoveContainer" containerID="d027eadc3c7bfbe1d14004061e72e5e80753c2ca290ccb53984e1d829ada50ac"
Oct 30 19:04:40.126135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:40.126100    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"]
Oct 30 19:04:40.175488 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:40.175444    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cfaf8925-575cbff66c-5xxvn"]
Oct 30 19:04:40.239963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:40.239868    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"]
Oct 30 19:04:40.244793 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:40.244759    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cfaf8925-7bb95b59d5-kwtmd"]
Oct 30 19:04:41.225223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:41.225150    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66" path="/var/lib/kubelet/pods/9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66/volumes"
Oct 30 19:04:41.225874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:41.225856    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a2d6fd-783b-4e85-b831-0ad295744988" path="/var/lib/kubelet/pods/c6a2d6fd-783b-4e85-b831-0ad295744988/volumes"
Oct 30 19:04:43.892411 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:43.892381    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:04:44.516775 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.516739    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"]
Oct 30 19:04:44.517439 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.517423    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c6a2d6fd-783b-4e85-b831-0ad295744988" containerName="payload-pre-processing"
Oct 30 19:04:44.517531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.517440    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a2d6fd-783b-4e85-b831-0ad295744988" containerName="payload-pre-processing"
Oct 30 19:04:44.517531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.517453    4267 cpu_manager.go:414] "containerMap: removing container" podUID="9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66" containerName="payload-processing"
Oct 30 19:04:44.517531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.517458    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66" containerName="payload-processing"
Oct 30 19:04:44.517640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.517575    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a2d6fd-783b-4e85-b831-0ad295744988" containerName="payload-pre-processing"
Oct 30 19:04:44.517640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.517584    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa5cbec-50c3-45a0-a6e0-8cfebbbc8d66" containerName="payload-processing"
Oct 30 19:04:44.522703 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.522686    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:44.553316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.553283    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:44.553566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.553346    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:44.553674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.553651    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z6jc\" (UniqueName: \"kubernetes.io/projected/963fcdf8-29ff-4996-9d46-57ab68ff225d-kube-api-access-6z6jc\") pod \"maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:44.576186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.576157    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:04:44.578692 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.578645    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"]
Oct 30 19:04:44.621185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.621152    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:04:44.655911 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.655875    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:44.656104 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.656072    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z6jc\" (UniqueName: \"kubernetes.io/projected/963fcdf8-29ff-4996-9d46-57ab68ff225d-kube-api-access-6z6jc\") pod \"maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:44.656288 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.656273    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:44.657164 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.657137    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:44.658742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.658722    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:44.715255 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.715214    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z6jc\" (UniqueName: \"kubernetes.io/projected/963fcdf8-29ff-4996-9d46-57ab68ff225d-kube-api-access-6z6jc\") pod \"maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:44.839659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:44.839572    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:45.222221 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:04:45.222187    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963fcdf8_29ff_4996_9d46_57ab68ff225d.slice/crio-3e204c32a5616b0e22955087b72feb9799ffd1cd15bf8b3bd9dc566328f633ba WatchSource:0}: Error finding container 3e204c32a5616b0e22955087b72feb9799ffd1cd15bf8b3bd9dc566328f633ba: Status 404 returned error can't find the container with id 3e204c32a5616b0e22955087b72feb9799ffd1cd15bf8b3bd9dc566328f633ba
Oct 30 19:04:45.226388 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:45.226360    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"]
Oct 30 19:04:46.013040 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:46.012995    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk" event={"ID":"963fcdf8-29ff-4996-9d46-57ab68ff225d","Type":"ContainerStarted","Data":"dbf1f987f66e1b0ef1702d2ab9d1cbf116b7cf523f9fccb1272c3cc1992ccc25"}
Oct 30 19:04:46.013241 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:46.013100    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk" event={"ID":"963fcdf8-29ff-4996-9d46-57ab68ff225d","Type":"ContainerStarted","Data":"3e204c32a5616b0e22955087b72feb9799ffd1cd15bf8b3bd9dc566328f633ba"}
Oct 30 19:04:46.130451 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:46.130391    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk" podStartSLOduration=2.130375726 podStartE2EDuration="2.130375726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:46.127539363 +0000 UTC m=+1577.015456453" watchObservedRunningTime="2026-10-30 19:04:46.130375726 +0000 UTC m=+1577.018292842"
Oct 30 19:04:47.024393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:47.024359    4267 generic.go:356] "Generic (PLEG): container finished" podID="963fcdf8-29ff-4996-9d46-57ab68ff225d" containerID="dbf1f987f66e1b0ef1702d2ab9d1cbf116b7cf523f9fccb1272c3cc1992ccc25" exitCode=0
Oct 30 19:04:47.024922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:47.024455    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk" event={"ID":"963fcdf8-29ff-4996-9d46-57ab68ff225d","Type":"ContainerDied","Data":"dbf1f987f66e1b0ef1702d2ab9d1cbf116b7cf523f9fccb1272c3cc1992ccc25"}
Oct 30 19:04:48.809993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:48.809948    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:48.918820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:48.917008    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-openshift-ca\") pod \"963fcdf8-29ff-4996-9d46-57ab68ff225d\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") "
Oct 30 19:04:48.918820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:48.917188    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963fcdf8-29ff-4996-9d46-57ab68ff225d-kube-api-access-6z6jc\" (UniqueName: \"kubernetes.io/projected/963fcdf8-29ff-4996-9d46-57ab68ff225d-kube-api-access-6z6jc\") pod \"963fcdf8-29ff-4996-9d46-57ab68ff225d\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") "
Oct 30 19:04:48.918820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:48.917380    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-xks-ca\") pod \"963fcdf8-29ff-4996-9d46-57ab68ff225d\" (UID: \"963fcdf8-29ff-4996-9d46-57ab68ff225d\") "
Oct 30 19:04:48.919152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:48.918889    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-openshift-ca" pod "963fcdf8-29ff-4996-9d46-57ab68ff225d" (UID: "963fcdf8-29ff-4996-9d46-57ab68ff225d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:48.922213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:48.921710    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963fcdf8-29ff-4996-9d46-57ab68ff225d-kube-api-access-6z6jc" pod "963fcdf8-29ff-4996-9d46-57ab68ff225d" (UID: "963fcdf8-29ff-4996-9d46-57ab68ff225d"). InnerVolumeSpecName "kube-api-access-6z6jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:48.923676 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:48.923619    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-xks-ca" pod "963fcdf8-29ff-4996-9d46-57ab68ff225d" (UID: "963fcdf8-29ff-4996-9d46-57ab68ff225d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:49.019367 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:49.019326    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:49.019367 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:49.019362    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z6jc\" (UniqueName: \"kubernetes.io/projected/963fcdf8-29ff-4996-9d46-57ab68ff225d-kube-api-access-6z6jc\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:49.019606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:49.019377    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963fcdf8-29ff-4996-9d46-57ab68ff225d-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:04:49.117152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:49.117124    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"
Oct 30 19:04:49.117355 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:49.117163    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk" event={"ID":"963fcdf8-29ff-4996-9d46-57ab68ff225d","Type":"ContainerDied","Data":"3e204c32a5616b0e22955087b72feb9799ffd1cd15bf8b3bd9dc566328f633ba"}
Oct 30 19:04:49.117355 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:49.117214    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e204c32a5616b0e22955087b72feb9799ffd1cd15bf8b3bd9dc566328f633ba"
Oct 30 19:04:50.159988 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:04:50.159946    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:04:55.960712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:55.960675    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"]
Oct 30 19:04:56.076798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:56.076727    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"]
Oct 30 19:04:58.064580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:58.064545    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"]
Oct 30 19:04:58.106428 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:58.106398    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1702ab-n7vnq"]
Oct 30 19:04:59.224672 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:04:59.224639    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b102a36-afea-450a-b0b7-d757364ebeda" path="/var/lib/kubelet/pods/5b102a36-afea-450a-b0b7-d757364ebeda/volumes"
Oct 30 19:05:00.960989 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:00.960946    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l" podUID="6e26ef57-61a1-4045-b15d-a692c4418de3" containerName="payload-processing" containerID="cri-o://7e54e9bb864c93b844a735bd74158a323e5bc9a44347906bb626a562a80635fb" gracePeriod=25
Oct 30 19:05:01.077812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.077770    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2" podUID="8b5988ce-d59e-4cb6-a637-997e33ba4541" containerName="payload-pre-processing" containerID="cri-o://af10653dab7844e585396727b188708d5b87df30e338b61311ad031d7b3c3a31" gracePeriod=25
Oct 30 19:05:01.253177 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.252922    4267 generic.go:356] "Generic (PLEG): container finished" podID="6e26ef57-61a1-4045-b15d-a692c4418de3" containerID="7e54e9bb864c93b844a735bd74158a323e5bc9a44347906bb626a562a80635fb" exitCode=0
Oct 30 19:05:01.253177 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.253056    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l" event={"ID":"6e26ef57-61a1-4045-b15d-a692c4418de3","Type":"ContainerDied","Data":"7e54e9bb864c93b844a735bd74158a323e5bc9a44347906bb626a562a80635fb"}
Oct 30 19:05:01.260915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.260687    4267 generic.go:356] "Generic (PLEG): container finished" podID="8b5988ce-d59e-4cb6-a637-997e33ba4541" containerID="af10653dab7844e585396727b188708d5b87df30e338b61311ad031d7b3c3a31" exitCode=0
Oct 30 19:05:01.260915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.260848    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2" event={"ID":"8b5988ce-d59e-4cb6-a637-997e33ba4541","Type":"ContainerDied","Data":"af10653dab7844e585396727b188708d5b87df30e338b61311ad031d7b3c3a31"}
Oct 30 19:05:01.318017 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.309178    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:05:01.386072 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.386043    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:05:01.431342 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.431311    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e26ef57-61a1-4045-b15d-a692c4418de3-kube-api-access-hs59z\" (UniqueName: \"kubernetes.io/projected/6e26ef57-61a1-4045-b15d-a692c4418de3-kube-api-access-hs59z\") pod \"6e26ef57-61a1-4045-b15d-a692c4418de3\" (UID: \"6e26ef57-61a1-4045-b15d-a692c4418de3\") "
Oct 30 19:05:01.431579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.431380    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e26ef57-61a1-4045-b15d-a692c4418de3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e26ef57-61a1-4045-b15d-a692c4418de3-plugins-config-volume\") pod \"6e26ef57-61a1-4045-b15d-a692c4418de3\" (UID: \"6e26ef57-61a1-4045-b15d-a692c4418de3\") "
Oct 30 19:05:01.431579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.431431    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5988ce-d59e-4cb6-a637-997e33ba4541-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5988ce-d59e-4cb6-a637-997e33ba4541-plugins-config-volume\") pod \"8b5988ce-d59e-4cb6-a637-997e33ba4541\" (UID: \"8b5988ce-d59e-4cb6-a637-997e33ba4541\") "
Oct 30 19:05:01.431707 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.431580    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5988ce-d59e-4cb6-a637-997e33ba4541-kube-api-access-cpfc6\" (UniqueName: \"kubernetes.io/projected/8b5988ce-d59e-4cb6-a637-997e33ba4541-kube-api-access-cpfc6\") pod \"8b5988ce-d59e-4cb6-a637-997e33ba4541\" (UID: \"8b5988ce-d59e-4cb6-a637-997e33ba4541\") "
Oct 30 19:05:01.431956 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.431927    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5988ce-d59e-4cb6-a637-997e33ba4541-plugins-config-volume" pod "8b5988ce-d59e-4cb6-a637-997e33ba4541" (UID: "8b5988ce-d59e-4cb6-a637-997e33ba4541"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:01.432222 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.432202    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5988ce-d59e-4cb6-a637-997e33ba4541-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:01.432387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.432365    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e26ef57-61a1-4045-b15d-a692c4418de3-plugins-config-volume" pod "6e26ef57-61a1-4045-b15d-a692c4418de3" (UID: "6e26ef57-61a1-4045-b15d-a692c4418de3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:01.434546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.434483    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e26ef57-61a1-4045-b15d-a692c4418de3-kube-api-access-hs59z" pod "6e26ef57-61a1-4045-b15d-a692c4418de3" (UID: "6e26ef57-61a1-4045-b15d-a692c4418de3"). InnerVolumeSpecName "kube-api-access-hs59z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:01.435026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.435002    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5988ce-d59e-4cb6-a637-997e33ba4541-kube-api-access-cpfc6" pod "8b5988ce-d59e-4cb6-a637-997e33ba4541" (UID: "8b5988ce-d59e-4cb6-a637-997e33ba4541"). InnerVolumeSpecName "kube-api-access-cpfc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:01.533335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.533248    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs59z\" (UniqueName: \"kubernetes.io/projected/6e26ef57-61a1-4045-b15d-a692c4418de3-kube-api-access-hs59z\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:01.533335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.533279    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e26ef57-61a1-4045-b15d-a692c4418de3-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:01.533335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:01.533289    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpfc6\" (UniqueName: \"kubernetes.io/projected/8b5988ce-d59e-4cb6-a637-997e33ba4541-kube-api-access-cpfc6\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:02.270137 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:02.270054    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"
Oct 30 19:05:02.270137 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:02.270089    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2" event={"ID":"8b5988ce-d59e-4cb6-a637-997e33ba4541","Type":"ContainerDied","Data":"7986590fb9579cba0bd290dd9f2935d0aba5aff07c23248d10e6044d0007713c"}
Oct 30 19:05:02.270623 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:02.270151    4267 scope.go:122] "RemoveContainer" containerID="af10653dab7844e585396727b188708d5b87df30e338b61311ad031d7b3c3a31"
Oct 30 19:05:02.271988 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:02.271957    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l" event={"ID":"6e26ef57-61a1-4045-b15d-a692c4418de3","Type":"ContainerDied","Data":"8b351c1326ebdd63180520465d35722775ec9782c6cdef452cdfe1655e679c1e"}
Oct 30 19:05:02.272072 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:02.271992    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"
Oct 30 19:05:02.283550 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:02.283529    4267 scope.go:122] "RemoveContainer" containerID="7e54e9bb864c93b844a735bd74158a323e5bc9a44347906bb626a562a80635fb"
Oct 30 19:05:02.443326 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:02.443288    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"]
Oct 30 19:05:02.489115 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:02.489072    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4b66bd06-86b89f456c-8f42l"]
Oct 30 19:05:02.564353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:02.564267    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"]
Oct 30 19:05:02.574859 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:02.574821    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4b66bd06-6cc878b775-dzmh2"]
Oct 30 19:05:03.226036 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:03.225999    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e26ef57-61a1-4045-b15d-a692c4418de3" path="/var/lib/kubelet/pods/6e26ef57-61a1-4045-b15d-a692c4418de3/volumes"
Oct 30 19:05:03.226710 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:03.226691    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b5988ce-d59e-4cb6-a637-997e33ba4541" path="/var/lib/kubelet/pods/8b5988ce-d59e-4cb6-a637-997e33ba4541/volumes"
Oct 30 19:05:07.107228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.107189    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"]
Oct 30 19:05:07.108146 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.108127    4267 cpu_manager.go:414] "containerMap: removing container" podUID="963fcdf8-29ff-4996-9d46-57ab68ff225d" containerName="revoke-keys"
Oct 30 19:05:07.108228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.108148    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963fcdf8-29ff-4996-9d46-57ab68ff225d" containerName="revoke-keys"
Oct 30 19:05:07.108228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.108165    4267 cpu_manager.go:414] "containerMap: removing container" podUID="6e26ef57-61a1-4045-b15d-a692c4418de3" containerName="payload-processing"
Oct 30 19:05:07.108228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.108172    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e26ef57-61a1-4045-b15d-a692c4418de3" containerName="payload-processing"
Oct 30 19:05:07.108228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.108197    4267 cpu_manager.go:414] "containerMap: removing container" podUID="8b5988ce-d59e-4cb6-a637-997e33ba4541" containerName="payload-pre-processing"
Oct 30 19:05:07.108228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.108205    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5988ce-d59e-4cb6-a637-997e33ba4541" containerName="payload-pre-processing"
Oct 30 19:05:07.108468 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.108331    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="963fcdf8-29ff-4996-9d46-57ab68ff225d" containerName="revoke-keys"
Oct 30 19:05:07.108468 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.108341    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e26ef57-61a1-4045-b15d-a692c4418de3" containerName="payload-processing"
Oct 30 19:05:07.108468 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.108348    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5988ce-d59e-4cb6-a637-997e33ba4541" containerName="payload-pre-processing"
Oct 30 19:05:07.113446 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.113426    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.170140 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.170115    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-36d676-serving-cert\""
Oct 30 19:05:07.218663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.218628    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.218826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.218762    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.218962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.218938    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.219119 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.219052    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khwws\" (UniqueName: \"kubernetes.io/projected/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-kube-api-access-khwws\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.228173 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.228144    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"]
Oct 30 19:05:07.320661 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.320626    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.320838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.320736    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khwws\" (UniqueName: \"kubernetes.io/projected/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-kube-api-access-khwws\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.320893 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.320869    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.321004 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.320988    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.321140 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:05:07.321121    4267 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-serving-cert: secret "maas-api-e2e-shared-a-w3-36d676-serving-cert" not found
Oct 30 19:05:07.321211 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:05:07.321192    4267 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-maas-api-tls podName:91d78de9-1580-4e7b-a3e3-e0a79f32dcf2 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:07.821175934 +0000 UTC m=+1598.709093006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-maas-api-tls") pod "maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d" (UID: "91d78de9-1580-4e7b-a3e3-e0a79f32dcf2") : secret "maas-api-e2e-shared-a-w3-36d676-serving-cert" not found
Oct 30 19:05:07.321389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.321371    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.323295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.323272    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.388405 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.388284    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khwws\" (UniqueName: \"kubernetes.io/projected/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-kube-api-access-khwws\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.835102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.835044    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:07.865529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:07.865279    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:08.027843 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:08.027796    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:08.492526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:08.491856    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"]
Oct 30 19:05:08.505046 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:05:08.504966    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d78de9_1580_4e7b_a3e3_e0a79f32dcf2.slice/crio-41d357f24a881ce975ec118f3bf998e06f085e0077ae210d0c90bafab483925e WatchSource:0}: Error finding container 41d357f24a881ce975ec118f3bf998e06f085e0077ae210d0c90bafab483925e: Status 404 returned error can't find the container with id 41d357f24a881ce975ec118f3bf998e06f085e0077ae210d0c90bafab483925e
Oct 30 19:05:09.345565 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:09.345524    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d" event={"ID":"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2","Type":"ContainerStarted","Data":"41d357f24a881ce975ec118f3bf998e06f085e0077ae210d0c90bafab483925e"}
Oct 30 19:05:10.378846 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:10.377534    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d" event={"ID":"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2","Type":"ContainerStarted","Data":"be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889"}
Oct 30 19:05:10.378846 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:10.378319    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:10.497398 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:10.497343    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d" podStartSLOduration=2.623202321 podStartE2EDuration="3.497329251s" totalImagesPullingTime="874.12693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:10.493807031 +0000 UTC m=+1601.381724117" watchObservedRunningTime="2026-10-30 19:05:10.497329251 +0000 UTC m=+1601.385246392"
Oct 30 19:05:16.390787 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:16.390756    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:05:20.183174 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:05:20.183115    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:05:27.590089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.590025    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"]
Oct 30 19:05:27.597888 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.597865    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:05:27.650787 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.650201    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"]
Oct 30 19:05:27.671273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.671234    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6222\" (UniqueName: \"kubernetes.io/projected/01a56c08-711a-43c4-b99e-f857cface646-kube-api-access-s6222\") pod \"payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r\" (UID: \"01a56c08-711a-43c4-b99e-f857cface646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:05:27.671899 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.671869    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a56c08-711a-43c4-b99e-f857cface646-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r\" (UID: \"01a56c08-711a-43c4-b99e-f857cface646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:05:27.698815 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.698781    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-eb4c249f\""
Oct 30 19:05:27.699184 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.699164    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-eb4c249f-dockercfg-kv74j\""
Oct 30 19:05:27.828519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.796687    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a56c08-711a-43c4-b99e-f857cface646-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r\" (UID: \"01a56c08-711a-43c4-b99e-f857cface646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:05:27.828918 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.803636    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a56c08-711a-43c4-b99e-f857cface646-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r\" (UID: \"01a56c08-711a-43c4-b99e-f857cface646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:05:27.829306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.829280    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6222\" (UniqueName: \"kubernetes.io/projected/01a56c08-711a-43c4-b99e-f857cface646-kube-api-access-s6222\") pod \"payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r\" (UID: \"01a56c08-711a-43c4-b99e-f857cface646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:05:27.901902 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.901787    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6222\" (UniqueName: \"kubernetes.io/projected/01a56c08-711a-43c4-b99e-f857cface646-kube-api-access-s6222\") pod \"payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r\" (UID: \"01a56c08-711a-43c4-b99e-f857cface646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:05:27.921543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:27.921508    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:05:28.265510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.265460    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"]
Oct 30 19:05:28.265729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.265702    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl" podUID="72cc29af-2da5-4227-a96c-42482a1e1f00" containerName="maas-api" containerID="cri-o://ec6c0a8b288a8ae86b870fe39f9568e20b3de280d48bc9e8cd27053193f4e729" gracePeriod=30
Oct 30 19:05:28.377891 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.377862    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"]
Oct 30 19:05:28.378025 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:05:28.377992    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a56c08_711a_43c4_b99e_f857cface646.slice/crio-c5e9432023600931dcfc67f8bed425b7f7e40849299d5b85c95eae1334fc122b WatchSource:0}: Error finding container c5e9432023600931dcfc67f8bed425b7f7e40849299d5b85c95eae1334fc122b: Status 404 returned error can't find the container with id c5e9432023600931dcfc67f8bed425b7f7e40849299d5b85c95eae1334fc122b
Oct 30 19:05:28.633550 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.633474    4267 generic.go:356] "Generic (PLEG): container finished" podID="72cc29af-2da5-4227-a96c-42482a1e1f00" containerID="ec6c0a8b288a8ae86b870fe39f9568e20b3de280d48bc9e8cd27053193f4e729" exitCode=0
Oct 30 19:05:28.634029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.633619    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl" event={"ID":"72cc29af-2da5-4227-a96c-42482a1e1f00","Type":"ContainerDied","Data":"ec6c0a8b288a8ae86b870fe39f9568e20b3de280d48bc9e8cd27053193f4e729"}
Oct 30 19:05:28.634310 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.634288    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:05:28.636075 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.636048    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r" event={"ID":"01a56c08-711a-43c4-b99e-f857cface646","Type":"ContainerStarted","Data":"10255b45112ff628b7ad5ebd0e20702e9ba5b8a8e783fcdd5a863361e1cb05c1"}
Oct 30 19:05:28.636269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.636251    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r" event={"ID":"01a56c08-711a-43c4-b99e-f857cface646","Type":"ContainerStarted","Data":"c5e9432023600931dcfc67f8bed425b7f7e40849299d5b85c95eae1334fc122b"}
Oct 30 19:05:28.636674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.636653    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:05:28.677636 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.677599    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-metrics-certs\") pod \"72cc29af-2da5-4227-a96c-42482a1e1f00\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") "
Oct 30 19:05:28.677811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.677645    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cc29af-2da5-4227-a96c-42482a1e1f00-kube-api-access-m4wwj\" (UniqueName: \"kubernetes.io/projected/72cc29af-2da5-4227-a96c-42482a1e1f00-kube-api-access-m4wwj\") pod \"72cc29af-2da5-4227-a96c-42482a1e1f00\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") "
Oct 30 19:05:28.677811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.677714    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-maas-api-tls\") pod \"72cc29af-2da5-4227-a96c-42482a1e1f00\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") "
Oct 30 19:05:28.677886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.677873    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72cc29af-2da5-4227-a96c-42482a1e1f00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cc29af-2da5-4227-a96c-42482a1e1f00-openshift-service-ca\") pod \"72cc29af-2da5-4227-a96c-42482a1e1f00\" (UID: \"72cc29af-2da5-4227-a96c-42482a1e1f00\") "
Oct 30 19:05:28.678251 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.678223    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cc29af-2da5-4227-a96c-42482a1e1f00-openshift-service-ca" pod "72cc29af-2da5-4227-a96c-42482a1e1f00" (UID: "72cc29af-2da5-4227-a96c-42482a1e1f00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:28.678946 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.678921    4267 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72cc29af-2da5-4227-a96c-42482a1e1f00-openshift-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:28.679913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.679890    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-metrics-certs" pod "72cc29af-2da5-4227-a96c-42482a1e1f00" (UID: "72cc29af-2da5-4227-a96c-42482a1e1f00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:28.680012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.679948    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-maas-api-tls" pod "72cc29af-2da5-4227-a96c-42482a1e1f00" (UID: "72cc29af-2da5-4227-a96c-42482a1e1f00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:28.680012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.679966    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cc29af-2da5-4227-a96c-42482a1e1f00-kube-api-access-m4wwj" pod "72cc29af-2da5-4227-a96c-42482a1e1f00" (UID: "72cc29af-2da5-4227-a96c-42482a1e1f00"). InnerVolumeSpecName "kube-api-access-m4wwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:28.790736 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.783253    4267 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-metrics-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:28.790736 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.783296    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4wwj\" (UniqueName: \"kubernetes.io/projected/72cc29af-2da5-4227-a96c-42482a1e1f00-kube-api-access-m4wwj\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:28.790736 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.783315    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72cc29af-2da5-4227-a96c-42482a1e1f00-maas-api-tls\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:05:28.896786 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:28.896679    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r" podStartSLOduration=1.8966659639999999 podStartE2EDuration="1.896665964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:28.892429547 +0000 UTC m=+1619.780346636" watchObservedRunningTime="2026-10-30 19:05:28.896665964 +0000 UTC m=+1619.784583055"
Oct 30 19:05:29.667589 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:29.667473    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"
Oct 30 19:05:29.668110 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:29.667744    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl" event={"ID":"72cc29af-2da5-4227-a96c-42482a1e1f00","Type":"ContainerDied","Data":"2b8aaa6a209fbe0c5e40e99267ae42d6f7373b00891c2cd7853a57c6fa8cc8ca"}
Oct 30 19:05:29.668110 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:29.667806    4267 scope.go:122] "RemoveContainer" containerID="ec6c0a8b288a8ae86b870fe39f9568e20b3de280d48bc9e8cd27053193f4e729"
Oct 30 19:05:29.835312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:29.835261    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"]
Oct 30 19:05:29.884665 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:29.884631    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cd57e7dd-5fb59b88d-rvjbl"]
Oct 30 19:05:30.433832 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:30.433791    4267 scope.go:122] "RemoveContainer" containerID="24d192343064371ec5bc2ac69a39b7d83f239c5ca898442fe3ad3ce708d5ae0f"
Oct 30 19:05:30.444972 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:30.444952    4267 scope.go:122] "RemoveContainer" containerID="fcef0a3c712c1674bce2c34e702390e15b65b870d6c3436d50bbaf3e10d73d1b"
Oct 30 19:05:31.235460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:31.235423    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cc29af-2da5-4227-a96c-42482a1e1f00" path="/var/lib/kubelet/pods/72cc29af-2da5-4227-a96c-42482a1e1f00/volumes"
Oct 30 19:05:37.546521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:37.546469    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"]
Oct 30 19:05:37.587928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:37.587885    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b66bd06-z7twk"]
Oct 30 19:05:39.227415 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:39.227373    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963fcdf8-29ff-4996-9d46-57ab68ff225d" path="/var/lib/kubelet/pods/963fcdf8-29ff-4996-9d46-57ab68ff225d/volumes"
Oct 30 19:05:39.671215 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:39.671137    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:05:43.827885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:43.827843    4267 generic.go:356] "Generic (PLEG): container finished" podID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerID="fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f" exitCode=0
Oct 30 19:05:43.828353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:43.828077    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" event={"ID":"5f6af68d-aabc-4ced-b194-395f48c4e36c","Type":"ContainerDied","Data":"fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f"}
Oct 30 19:05:43.828648 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:43.828623    4267 scope.go:122] "RemoveContainer" containerID="fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f"
Oct 30 19:05:43.830385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:43.830357    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:05:43.830462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:43.830430    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:05:43.830525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:43.830467    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:05:44.874733 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:44.874696    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" event={"ID":"5f6af68d-aabc-4ced-b194-395f48c4e36c","Type":"ContainerStarted","Data":"93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f"}
Oct 30 19:05:45.169481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:45.169403    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:05:45.169678 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:45.169590    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:05:45.169678 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:45.169653    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:05:46.169479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:46.169443    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:05:46.169927 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:46.169516    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:05:47.169110 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:47.169076    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:05:47.169370 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:47.169165    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:05:48.169519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:48.169354    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:05:48.169519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:48.169413    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:05:49.169190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:49.169148    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:05:49.169514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:49.169462    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:05:50.165651 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:05:50.165582    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:05:50.170026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:50.169998    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:05:50.170264 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:50.170217    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 19:05:51.174472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:51.174436    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:05:51.174982 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:51.174955    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:05:51.175629 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:51.175608    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:05:58.053241 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:58.053184    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"]
Oct 30 19:05:58.100059 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:58.100024    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aba1b03-2mlhs"]
Oct 30 19:05:59.229159 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:05:59.229115    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ad2ed9-30cc-4e03-9bd9-5d358127aef5" path="/var/lib/kubelet/pods/75ad2ed9-30cc-4e03-9bd9-5d358127aef5/volumes"
Oct 30 19:06:05.072541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:05.072505    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"]
Oct 30 19:06:05.122609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:05.122573    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a3abf53-86vkw"]
Oct 30 19:06:05.222788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:05.222745    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f3bd02-6ba1-420a-add2-e2068d1796ca" path="/var/lib/kubelet/pods/e3f3bd02-6ba1-420a-add2-e2068d1796ca/volumes"
Oct 30 19:06:09.049488 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:09.049452    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"]
Oct 30 19:06:09.098227 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:09.098195    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ae76b8-kvxsd"]
Oct 30 19:06:09.241613 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:09.241580    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85" path="/var/lib/kubelet/pods/3dcbaa21-bc20-4fc9-ba01-68d2dcff5c85/volumes"
Oct 30 19:06:17.190336 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.190300    4267 generic.go:356] "Generic (PLEG): container finished" podID="41f3d4ae-3622-413b-9038-debcc28bec24" containerID="ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44" exitCode=0
Oct 30 19:06:17.190908 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.190372    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" event={"ID":"41f3d4ae-3622-413b-9038-debcc28bec24","Type":"ContainerDied","Data":"ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44"}
Oct 30 19:06:17.190969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.190932    4267 scope.go:122] "RemoveContainer" containerID="ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44"
Oct 30 19:06:17.192810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.192778    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:06:17.192952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.192869    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:06:17.192952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.192920    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:06:17.442125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.442090    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"]
Oct 30 19:06:17.442884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.442860    4267 cpu_manager.go:414] "containerMap: removing container" podUID="72cc29af-2da5-4227-a96c-42482a1e1f00" containerName="maas-api"
Oct 30 19:06:17.442884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.442882    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cc29af-2da5-4227-a96c-42482a1e1f00" containerName="maas-api"
Oct 30 19:06:17.443126 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.443111    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cc29af-2da5-4227-a96c-42482a1e1f00" containerName="maas-api"
Oct 30 19:06:17.446615 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.446595    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.472728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.472690    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e14a47-d527-4fc9-b376-ada263d9d5eb-openshift-service-ca\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.472918 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.472798    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.472981 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.472917    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcf7\" (UniqueName: \"kubernetes.io/projected/39e14a47-d527-4fc9-b376-ada263d9d5eb-kube-api-access-pbcf7\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.473035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.472991    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-metrics-certs\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.505947 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.505862    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-eb4c249f-serving-cert\""
Oct 30 19:06:17.571392 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.571353    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"]
Oct 30 19:06:17.574482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.574450    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcf7\" (UniqueName: \"kubernetes.io/projected/39e14a47-d527-4fc9-b376-ada263d9d5eb-kube-api-access-pbcf7\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.575112 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.575084    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-metrics-certs\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.575464 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.575441    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e14a47-d527-4fc9-b376-ada263d9d5eb-openshift-service-ca\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.575647 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.575629    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.575967 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:06:17.575949    4267 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-serving-cert: secret "maas-api-e2e-ait-eb4c249f-serving-cert" not found
Oct 30 19:06:17.576140 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:06:17.576126    4267 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls podName:39e14a47-d527-4fc9-b376-ada263d9d5eb nodeName:}" failed. No retries permitted until 2026-10-30 19:06:18.076103167 +0000 UTC m=+1668.964020239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls") pod "maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s" (UID: "39e14a47-d527-4fc9-b376-ada263d9d5eb") : secret "maas-api-e2e-ait-eb4c249f-serving-cert" not found
Oct 30 19:06:17.583789 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.583761    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e14a47-d527-4fc9-b376-ada263d9d5eb-openshift-service-ca\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.587382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.587359    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-metrics-certs\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:17.633332 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:17.633302    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcf7\" (UniqueName: \"kubernetes.io/projected/39e14a47-d527-4fc9-b376-ada263d9d5eb-kube-api-access-pbcf7\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:18.091142 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:18.091105    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:18.091366 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:06:18.091271    4267 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-serving-cert: secret "maas-api-e2e-ait-eb4c249f-serving-cert" not found
Oct 30 19:06:18.091366 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:06:18.091353    4267 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls podName:39e14a47-d527-4fc9-b376-ada263d9d5eb nodeName:}" failed. No retries permitted until 2026-10-30 19:06:19.09133511 +0000 UTC m=+1669.979252205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls") pod "maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s" (UID: "39e14a47-d527-4fc9-b376-ada263d9d5eb") : secret "maas-api-e2e-ait-eb4c249f-serving-cert" not found
Oct 30 19:06:18.202683 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:18.202646    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" event={"ID":"41f3d4ae-3622-413b-9038-debcc28bec24","Type":"ContainerStarted","Data":"09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816"}
Oct 30 19:06:18.847745 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:18.847710    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:06:18.847929 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:18.847900    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:06:18.847974 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:18.847944    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:06:19.107524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:19.107420    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:19.139484 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:19.138194    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls\") pod \"maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:19.265376 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:19.265341    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:19.691583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:19.691551    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"]
Oct 30 19:06:19.693975 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:06:19.693952    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e14a47_d527_4fc9_b376_ada263d9d5eb.slice/crio-aa2cb5f553df6792af7fb5c179cbdaf6b1303547c115245fe2bb9cba919af1dd WatchSource:0}: Error finding container aa2cb5f553df6792af7fb5c179cbdaf6b1303547c115245fe2bb9cba919af1dd: Status 404 returned error can't find the container with id aa2cb5f553df6792af7fb5c179cbdaf6b1303547c115245fe2bb9cba919af1dd
Oct 30 19:06:19.848578 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:19.848540    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:06:19.848769 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:19.848600    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:06:20.168316 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:06:20.168222    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:06:20.246561 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:20.246486    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s" event={"ID":"39e14a47-d527-4fc9-b376-ada263d9d5eb","Type":"ContainerStarted","Data":"aa2cb5f553df6792af7fb5c179cbdaf6b1303547c115245fe2bb9cba919af1dd"}
Oct 30 19:06:20.848122 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:20.848018    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:06:20.848122 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:20.848087    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:06:21.897520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:21.880142    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:06:21.897520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:21.880405    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:06:21.944519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:21.939616    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"]
Oct 30 19:06:22.004460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:21.999203    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:22.004460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:21.999252    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"]
Oct 30 19:06:22.004460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:21.999284    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s" event={"ID":"39e14a47-d527-4fc9-b376-ada263d9d5eb","Type":"ContainerStarted","Data":"c84c90b2b9b864440194a9548498a6d12df4a436fb790eb24d28c486911137ef"}
Oct 30 19:06:22.004460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:21.999335    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:06:22.056480 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.056431    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s" podStartSLOduration=4.157402216 podStartE2EDuration="5.05641608s" totalImagesPullingTime="899.013864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:22.007190558 +0000 UTC m=+1672.895107648" watchObservedRunningTime="2026-10-30 19:06:22.05641608 +0000 UTC m=+1672.944333169"
Oct 30 19:06:22.059866 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.059838    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-732a3f-dockercfg-sbd79\""
Oct 30 19:06:22.064072 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.063981    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"]
Oct 30 19:06:22.064807 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.064752    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-732a3f\""
Oct 30 19:06:22.070792 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.070770    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5412588-bb2x4"]
Oct 30 19:06:22.105544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.105504    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9jd\" (UniqueName: \"kubernetes.io/projected/f2c2c888-6a51-447e-aaa9-33a834dcf653-kube-api-access-vk9jd\") pod \"payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s\" (UID: \"f2c2c888-6a51-447e-aaa9-33a834dcf653\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:06:22.105902 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.105876    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c2c888-6a51-447e-aaa9-33a834dcf653-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s\" (UID: \"f2c2c888-6a51-447e-aaa9-33a834dcf653\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:06:22.207174 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.207144    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9jd\" (UniqueName: \"kubernetes.io/projected/f2c2c888-6a51-447e-aaa9-33a834dcf653-kube-api-access-vk9jd\") pod \"payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s\" (UID: \"f2c2c888-6a51-447e-aaa9-33a834dcf653\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:06:22.207373 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.207335    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c2c888-6a51-447e-aaa9-33a834dcf653-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s\" (UID: \"f2c2c888-6a51-447e-aaa9-33a834dcf653\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:06:22.208121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.208099    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c2c888-6a51-447e-aaa9-33a834dcf653-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s\" (UID: \"f2c2c888-6a51-447e-aaa9-33a834dcf653\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:06:22.264909 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.264879    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9jd\" (UniqueName: \"kubernetes.io/projected/f2c2c888-6a51-447e-aaa9-33a834dcf653-kube-api-access-vk9jd\") pod \"payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s\" (UID: \"f2c2c888-6a51-447e-aaa9-33a834dcf653\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:06:22.358991 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.358956    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:06:22.771378 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:06:22.771346    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c2c888_6a51_447e_aaa9_33a834dcf653.slice/crio-e8f30bf0135025cf34eff60652e1714bab7633c2aa72d49810b846c43e2dac66 WatchSource:0}: Error finding container e8f30bf0135025cf34eff60652e1714bab7633c2aa72d49810b846c43e2dac66: Status 404 returned error can't find the container with id e8f30bf0135025cf34eff60652e1714bab7633c2aa72d49810b846c43e2dac66
Oct 30 19:06:22.771623 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.771586    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"]
Oct 30 19:06:22.848032 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.847994    4267 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 19:06:22.848166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.848053    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 19:06:22.888194 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:22.888158    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s" event={"ID":"f2c2c888-6a51-447e-aaa9-33a834dcf653","Type":"ContainerStarted","Data":"e8f30bf0135025cf34eff60652e1714bab7633c2aa72d49810b846c43e2dac66"}
Oct 30 19:06:23.274211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:23.274175    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749" path="/var/lib/kubelet/pods/a18c7bdf-6ce0-4e0a-af52-a0db4ea7e749/volumes"
Oct 30 19:06:23.852650 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:23.852619    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:06:23.853170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:23.853148    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:06:23.853815 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:23.853791    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:06:23.897835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:23.897807    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s" event={"ID":"f2c2c888-6a51-447e-aaa9-33a834dcf653","Type":"ContainerStarted","Data":"7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef"}
Oct 30 19:06:23.897925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:23.897891    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:06:24.203622 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:24.203554    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s" podStartSLOduration=3.203536252 podStartE2EDuration="3.203536252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:24.201950244 +0000 UTC m=+1675.089867338" watchObservedRunningTime="2026-10-30 19:06:24.203536252 +0000 UTC m=+1675.091453342"
Oct 30 19:06:27.055037 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:27.055002    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"]
Oct 30 19:06:27.104398 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:27.104366    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be6cbb63-kk4k2"]
Oct 30 19:06:27.224060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:27.224022    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b1f8d9-ab15-4999-bd24-9990568778fd" path="/var/lib/kubelet/pods/84b1f8d9-ab15-4999-bd24-9990568778fd/volumes"
Oct 30 19:06:27.895324 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:27.895291    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:06:28.083384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:28.083342    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"]
Oct 30 19:06:28.094049 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:28.094003    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48f7e613-bpw7m"]
Oct 30 19:06:29.225845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:29.225808    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5a987b-e25c-4832-9921-7ed50b5321d4" path="/var/lib/kubelet/pods/4c5a987b-e25c-4832-9921-7ed50b5321d4/volumes"
Oct 30 19:06:30.505345 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:30.505312    4267 scope.go:122] "RemoveContainer" containerID="05b1918547bc14c227c6e8123d3ceb3900660f26db4eed01d99338933593a0fd"
Oct 30 19:06:34.459570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.459527    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"]
Oct 30 19:06:34.467178 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.467139    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:06:34.515556 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.515478    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtk7m\" (UniqueName: \"kubernetes.io/projected/ddf30c94-65c8-433c-9de7-7c3d5deb1719-kube-api-access-gtk7m\") pod \"payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n\" (UID: \"ddf30c94-65c8-433c-9de7-7c3d5deb1719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:06:34.515767 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.515646    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf30c94-65c8-433c-9de7-7c3d5deb1719-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n\" (UID: \"ddf30c94-65c8-433c-9de7-7c3d5deb1719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:06:34.527017 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.526595    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"]
Oct 30 19:06:34.572811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.572773    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4501c8d4\""
Oct 30 19:06:34.626666 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.626630    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4501c8d4-dockercfg-2765p\""
Oct 30 19:06:34.640541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.631814    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf30c94-65c8-433c-9de7-7c3d5deb1719-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n\" (UID: \"ddf30c94-65c8-433c-9de7-7c3d5deb1719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:06:34.641868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.638033    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf30c94-65c8-433c-9de7-7c3d5deb1719-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n\" (UID: \"ddf30c94-65c8-433c-9de7-7c3d5deb1719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:06:34.642781 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.642215    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtk7m\" (UniqueName: \"kubernetes.io/projected/ddf30c94-65c8-433c-9de7-7c3d5deb1719-kube-api-access-gtk7m\") pod \"payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n\" (UID: \"ddf30c94-65c8-433c-9de7-7c3d5deb1719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:06:34.663038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.662996    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"]
Oct 30 19:06:34.679790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.679757    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"]
Oct 30 19:06:34.680115 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.680094    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:06:34.708421 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.708357    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtk7m\" (UniqueName: \"kubernetes.io/projected/ddf30c94-65c8-433c-9de7-7c3d5deb1719-kube-api-access-gtk7m\") pod \"payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n\" (UID: \"ddf30c94-65c8-433c-9de7-7c3d5deb1719\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:06:34.751217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.750555    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52394b87-7cd4-4084-bc08-8e4e097173c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5\" (UID: \"52394b87-7cd4-4084-bc08-8e4e097173c8\") " pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:06:34.751217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.750753    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm84q\" (UniqueName: \"kubernetes.io/projected/52394b87-7cd4-4084-bc08-8e4e097173c8-kube-api-access-mm84q\") pod \"payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5\" (UID: \"52394b87-7cd4-4084-bc08-8e4e097173c8\") " pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:06:34.786447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.786405    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:06:34.853424 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.853384    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52394b87-7cd4-4084-bc08-8e4e097173c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5\" (UID: \"52394b87-7cd4-4084-bc08-8e4e097173c8\") " pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:06:34.853697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.853537    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm84q\" (UniqueName: \"kubernetes.io/projected/52394b87-7cd4-4084-bc08-8e4e097173c8-kube-api-access-mm84q\") pod \"payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5\" (UID: \"52394b87-7cd4-4084-bc08-8e4e097173c8\") " pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:06:34.854399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.854371    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52394b87-7cd4-4084-bc08-8e4e097173c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5\" (UID: \"52394b87-7cd4-4084-bc08-8e4e097173c8\") " pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:06:34.995641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:34.979252    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:06:35.012727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:35.011558    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm84q\" (UniqueName: \"kubernetes.io/projected/52394b87-7cd4-4084-bc08-8e4e097173c8-kube-api-access-mm84q\") pod \"payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5\" (UID: \"52394b87-7cd4-4084-bc08-8e4e097173c8\") " pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:06:35.038055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:35.037372    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:06:35.263708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:35.263099    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"]
Oct 30 19:06:35.606428 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:06:35.606390    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52394b87_7cd4_4084_bc08_8e4e097173c8.slice/crio-4068a6ee42b3acc44ecf1b277417329aae454eb67f504b706372c70540eec186 WatchSource:0}: Error finding container 4068a6ee42b3acc44ecf1b277417329aae454eb67f504b706372c70540eec186: Status 404 returned error can't find the container with id 4068a6ee42b3acc44ecf1b277417329aae454eb67f504b706372c70540eec186
Oct 30 19:06:35.607383 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:35.607362    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"]
Oct 30 19:06:36.066025 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:36.065971    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n" event={"ID":"ddf30c94-65c8-433c-9de7-7c3d5deb1719","Type":"ContainerStarted","Data":"2724c0237e350e9cf0a6d280d2390b4b3fbd103f0da49eb68ee0dd6f773edc93"}
Oct 30 19:06:36.066224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:36.066092    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:06:36.066224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:36.066120    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n" event={"ID":"ddf30c94-65c8-433c-9de7-7c3d5deb1719","Type":"ContainerStarted","Data":"97dad9bd84498f2c1da6218572fa75f371c03eb9f7ed188df36c3a89b2561fb6"}
Oct 30 19:06:36.068162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:36.068130    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5" event={"ID":"52394b87-7cd4-4084-bc08-8e4e097173c8","Type":"ContainerStarted","Data":"86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e"}
Oct 30 19:06:36.068311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:36.068179    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5" event={"ID":"52394b87-7cd4-4084-bc08-8e4e097173c8","Type":"ContainerStarted","Data":"4068a6ee42b3acc44ecf1b277417329aae454eb67f504b706372c70540eec186"}
Oct 30 19:06:36.068311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:36.068200    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:06:36.074651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:36.074621    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"]
Oct 30 19:06:36.079441 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:36.079412    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704d3027-qtmnb"]
Oct 30 19:06:36.189894 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:36.189837    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n" podStartSLOduration=2.189823277 podStartE2EDuration="2.189823277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:36.18598646 +0000 UTC m=+1687.073903549" watchObservedRunningTime="2026-10-30 19:06:36.189823277 +0000 UTC m=+1687.077740366"
Oct 30 19:06:36.332569 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:36.331842    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5" podStartSLOduration=2.33182007 podStartE2EDuration="2.33182007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:36.320812274 +0000 UTC m=+1687.208729365" watchObservedRunningTime="2026-10-30 19:06:36.33182007 +0000 UTC m=+1687.219737162"
Oct 30 19:06:37.225663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:37.225628    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b9780b-61d8-4ce2-a151-219c29e5d967" path="/var/lib/kubelet/pods/f2b9780b-61d8-4ce2-a151-219c29e5d967/volumes"
Oct 30 19:06:39.667512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.661708    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"]
Oct 30 19:06:39.682532 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.682465    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"]
Oct 30 19:06:39.687643 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.687616    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:06:39.742270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.742239    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ae6a138d\""
Oct 30 19:06:39.818700 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.818667    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ae6a138d-dockercfg-mxcpd\""
Oct 30 19:06:39.819886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.819848    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt7fj\" (UniqueName: \"kubernetes.io/projected/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-kube-api-access-rt7fj\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq\" (UID: \"482f2d86-6da5-48b0-a8ff-52f3db5be0f1\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:06:39.820119 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.820097    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq\" (UID: \"482f2d86-6da5-48b0-a8ff-52f3db5be0f1\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:06:39.921938 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.921849    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt7fj\" (UniqueName: \"kubernetes.io/projected/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-kube-api-access-rt7fj\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq\" (UID: \"482f2d86-6da5-48b0-a8ff-52f3db5be0f1\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:06:39.922365 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.922339    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq\" (UID: \"482f2d86-6da5-48b0-a8ff-52f3db5be0f1\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:06:39.923458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.923436    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq\" (UID: \"482f2d86-6da5-48b0-a8ff-52f3db5be0f1\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:06:39.980385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:39.980352    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt7fj\" (UniqueName: \"kubernetes.io/projected/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-kube-api-access-rt7fj\") pod \"payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq\" (UID: \"482f2d86-6da5-48b0-a8ff-52f3db5be0f1\") " pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:06:40.011770 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:40.011729    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:06:40.429670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:40.429124    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"]
Oct 30 19:06:40.432288 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:06:40.432257    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482f2d86_6da5_48b0_a8ff_52f3db5be0f1.slice/crio-e385e1ccb01e3ef1326a322bcc257a24945953756e5b4418f2c15e7c390d6e91 WatchSource:0}: Error finding container e385e1ccb01e3ef1326a322bcc257a24945953756e5b4418f2c15e7c390d6e91: Status 404 returned error can't find the container with id e385e1ccb01e3ef1326a322bcc257a24945953756e5b4418f2c15e7c390d6e91
Oct 30 19:06:41.189541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:41.188482    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq" event={"ID":"482f2d86-6da5-48b0-a8ff-52f3db5be0f1","Type":"ContainerStarted","Data":"c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775"}
Oct 30 19:06:41.189541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:41.188662    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq" event={"ID":"482f2d86-6da5-48b0-a8ff-52f3db5be0f1","Type":"ContainerStarted","Data":"e385e1ccb01e3ef1326a322bcc257a24945953756e5b4418f2c15e7c390d6e91"}
Oct 30 19:06:41.189541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:41.188860    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:06:41.304772 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:41.304719    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq" podStartSLOduration=2.304703366 podStartE2EDuration="2.304703366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:41.303106104 +0000 UTC m=+1692.191023215" watchObservedRunningTime="2026-10-30 19:06:41.304703366 +0000 UTC m=+1692.192620454"
Oct 30 19:06:41.944857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:41.944827    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"]
Oct 30 19:06:41.950773 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:41.950740    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:06:42.006383 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.006349    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-87ed12\""
Oct 30 19:06:42.025547 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.024991    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"]
Oct 30 19:06:42.185519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.180184    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hdt7\" (UniqueName: \"kubernetes.io/projected/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-kube-api-access-5hdt7\") pod \"payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75\" (UID: \"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:06:42.185519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.180337    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75\" (UID: \"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:06:42.185519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.181035    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-87ed12-dockercfg-h8f6p\""
Oct 30 19:06:42.333919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.316213    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdt7\" (UniqueName: \"kubernetes.io/projected/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-kube-api-access-5hdt7\") pod \"payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75\" (UID: \"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:06:42.333919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.319125    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75\" (UID: \"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:06:42.333919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.322384    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75\" (UID: \"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:06:42.349623 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.349590    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"]
Oct 30 19:06:42.355639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.355607    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:06:42.356922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.356888    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"]
Oct 30 19:06:42.379320 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.379281    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdt7\" (UniqueName: \"kubernetes.io/projected/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-kube-api-access-5hdt7\") pod \"payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75\" (UID: \"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:06:42.421609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.421570    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bwb\" (UniqueName: \"kubernetes.io/projected/3618fa65-b410-4f76-af60-9e6dd0140f04-kube-api-access-94bwb\") pod \"payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9\" (UID: \"3618fa65-b410-4f76-af60-9e6dd0140f04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:06:42.421792 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.421756    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3618fa65-b410-4f76-af60-9e6dd0140f04-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9\" (UID: \"3618fa65-b410-4f76-af60-9e6dd0140f04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:06:42.523400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.523355    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bwb\" (UniqueName: \"kubernetes.io/projected/3618fa65-b410-4f76-af60-9e6dd0140f04-kube-api-access-94bwb\") pod \"payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9\" (UID: \"3618fa65-b410-4f76-af60-9e6dd0140f04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:06:42.523637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.523562    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3618fa65-b410-4f76-af60-9e6dd0140f04-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9\" (UID: \"3618fa65-b410-4f76-af60-9e6dd0140f04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:06:42.524563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.524537    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3618fa65-b410-4f76-af60-9e6dd0140f04-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9\" (UID: \"3618fa65-b410-4f76-af60-9e6dd0140f04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:06:42.570171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.570126    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:06:42.580973 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.580942    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bwb\" (UniqueName: \"kubernetes.io/projected/3618fa65-b410-4f76-af60-9e6dd0140f04-kube-api-access-94bwb\") pod \"payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9\" (UID: \"3618fa65-b410-4f76-af60-9e6dd0140f04\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:06:42.673807 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.673775    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:06:42.952942 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:06:42.952911    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681a58de_ae69_4ea5_a6c0_4ee14aa3a3ef.slice/crio-0c6cf2bab1e848e747d0f3e1b3777d6bf886238e19b001541e48260350ade357 WatchSource:0}: Error finding container 0c6cf2bab1e848e747d0f3e1b3777d6bf886238e19b001541e48260350ade357: Status 404 returned error can't find the container with id 0c6cf2bab1e848e747d0f3e1b3777d6bf886238e19b001541e48260350ade357
Oct 30 19:06:42.956669 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:42.956634    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"]
Oct 30 19:06:43.053148 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.053116    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"]
Oct 30 19:06:43.057720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.057693    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a46943a-sb8f9"]
Oct 30 19:06:43.080715 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.080689    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"]
Oct 30 19:06:43.228795 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.228713    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee017bb-6870-45aa-baf0-933740e18bf5" path="/var/lib/kubelet/pods/cee017bb-6870-45aa-baf0-933740e18bf5/volumes"
Oct 30 19:06:43.370462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.370418    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9" event={"ID":"3618fa65-b410-4f76-af60-9e6dd0140f04","Type":"ContainerStarted","Data":"2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925"}
Oct 30 19:06:43.370938 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.370484    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9" event={"ID":"3618fa65-b410-4f76-af60-9e6dd0140f04","Type":"ContainerStarted","Data":"e0af0bb05224c1c2eee1da7b9383d33e7231f5f6e05d99cf2c8dae7eb2510952"}
Oct 30 19:06:43.370938 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.370666    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:06:43.372275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.372246    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75" event={"ID":"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef","Type":"ContainerStarted","Data":"7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074"}
Oct 30 19:06:43.372421 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.372297    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:06:43.372421 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.372315    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75" event={"ID":"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef","Type":"ContainerStarted","Data":"0c6cf2bab1e848e747d0f3e1b3777d6bf886238e19b001541e48260350ade357"}
Oct 30 19:06:43.488927 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.488807    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9" podStartSLOduration=1.488785139 podStartE2EDuration="1.488785139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:43.484283284 +0000 UTC m=+1694.372200373" watchObservedRunningTime="2026-10-30 19:06:43.488785139 +0000 UTC m=+1694.376702229"
Oct 30 19:06:43.601026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:43.600973    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75" podStartSLOduration=2.600960025 podStartE2EDuration="2.600960025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:43.598518921 +0000 UTC m=+1694.486436012" watchObservedRunningTime="2026-10-30 19:06:43.600960025 +0000 UTC m=+1694.488877155"
Oct 30 19:06:47.080329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:47.080292    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:06:47.080845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:47.080382    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:06:50.166144 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:06:50.166092    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:06:52.364886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:52.364824    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:06:54.390226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:54.390192    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:06:54.390674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:06:54.390461    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:07:04.052525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:04.052469    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"]
Oct 30 19:07:04.101762 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:04.101719    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-098a255d-zthg7"]
Oct 30 19:07:05.222908 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:05.222873    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be3e767-d8fc-4301-a64b-76c2c360f428" path="/var/lib/kubelet/pods/9be3e767-d8fc-4301-a64b-76c2c360f428/volumes"
Oct 30 19:07:14.055952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:14.055896    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"]
Oct 30 19:07:14.101864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:14.101814    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-797c813d-2kgqg"]
Oct 30 19:07:15.225889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:15.225850    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5" path="/var/lib/kubelet/pods/a62447b2-8fb9-4f32-a8cb-1e8a57bd2ea5/volumes"
Oct 30 19:07:20.163808 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:07:20.163764    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:07:30.567063 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:30.567026    4267 scope.go:122] "RemoveContainer" containerID="46ad484d67b1464e1560164284537233f08c984b73d4d688091acb260df1e8cf"
Oct 30 19:07:30.578896 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:30.578870    4267 scope.go:122] "RemoveContainer" containerID="e446950d7e87c9c1462c1635a465f215541641ec5dbaa9bd604d56ef2d86c88b"
Oct 30 19:07:30.589571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:30.589551    4267 scope.go:122] "RemoveContainer" containerID="494e391086fc96cd2ef09cf1ee710ed62c9c47cc855744ddc20f0bc55d6beee5"
Oct 30 19:07:30.599660 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:30.599641    4267 scope.go:122] "RemoveContainer" containerID="08d626924e30055dd6ccd93e7d57a4509f9126c896f40a4bff9e7cdfe01954c8"
Oct 30 19:07:30.610004 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:30.609980    4267 scope.go:122] "RemoveContainer" containerID="9d17d3db0762850a2e7c71c9fa0b95f3930801568cd7c0911ded0642d873fc6d"
Oct 30 19:07:30.619864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:30.619844    4267 scope.go:122] "RemoveContainer" containerID="3803572ee900abf0b45f00c571b6f2ff37e600e49ed573ef98ef94697e4e220b"
Oct 30 19:07:36.048583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:36.048547    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"]
Oct 30 19:07:36.098272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:36.098238    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f9e7b1-9w4bf"]
Oct 30 19:07:37.274530 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:37.271441    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c5dc0d-17ca-4227-b009-cf8fca9cd182" path="/var/lib/kubelet/pods/20c5dc0d-17ca-4227-b009-cf8fca9cd182/volumes"
Oct 30 19:07:44.111680 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:44.108379    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"]
Oct 30 19:07:44.146867 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:44.146831    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a495d43-lfdkn"]
Oct 30 19:07:45.264137 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:45.264099    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee" path="/var/lib/kubelet/pods/a7f3ccd6-f8e9-47bb-aa0e-c839ef0ad9ee/volumes"
Oct 30 19:07:46.187254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:46.187215    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"]
Oct 30 19:07:50.159840 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:07:50.159797    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:07:51.188583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:51.188535    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r" podUID="01a56c08-711a-43c4-b99e-f857cface646" containerName="payload-pre-processing" containerID="cri-o://10255b45112ff628b7ad5ebd0e20702e9ba5b8a8e783fcdd5a863361e1cb05c1" gracePeriod=25
Oct 30 19:07:51.326695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:51.326664    4267 generic.go:356] "Generic (PLEG): container finished" podID="01a56c08-711a-43c4-b99e-f857cface646" containerID="10255b45112ff628b7ad5ebd0e20702e9ba5b8a8e783fcdd5a863361e1cb05c1" exitCode=0
Oct 30 19:07:51.326889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:51.326735    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r" event={"ID":"01a56c08-711a-43c4-b99e-f857cface646","Type":"ContainerDied","Data":"10255b45112ff628b7ad5ebd0e20702e9ba5b8a8e783fcdd5a863361e1cb05c1"}
Oct 30 19:07:51.509427 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:51.509402    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:07:51.546353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:51.546312    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a56c08-711a-43c4-b99e-f857cface646-kube-api-access-s6222\" (UniqueName: \"kubernetes.io/projected/01a56c08-711a-43c4-b99e-f857cface646-kube-api-access-s6222\") pod \"01a56c08-711a-43c4-b99e-f857cface646\" (UID: \"01a56c08-711a-43c4-b99e-f857cface646\") "
Oct 30 19:07:51.546589 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:51.546400    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a56c08-711a-43c4-b99e-f857cface646-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a56c08-711a-43c4-b99e-f857cface646-plugins-config-volume\") pod \"01a56c08-711a-43c4-b99e-f857cface646\" (UID: \"01a56c08-711a-43c4-b99e-f857cface646\") "
Oct 30 19:07:51.546832 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:51.546804    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a56c08-711a-43c4-b99e-f857cface646-plugins-config-volume" pod "01a56c08-711a-43c4-b99e-f857cface646" (UID: "01a56c08-711a-43c4-b99e-f857cface646"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:51.547272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:51.547249    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a56c08-711a-43c4-b99e-f857cface646-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:07:51.548699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:51.548676    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a56c08-711a-43c4-b99e-f857cface646-kube-api-access-s6222" pod "01a56c08-711a-43c4-b99e-f857cface646" (UID: "01a56c08-711a-43c4-b99e-f857cface646"). InnerVolumeSpecName "kube-api-access-s6222". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:51.648923 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:51.648882    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6222\" (UniqueName: \"kubernetes.io/projected/01a56c08-711a-43c4-b99e-f857cface646-kube-api-access-s6222\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:07:52.336807 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:52.336769    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r" event={"ID":"01a56c08-711a-43c4-b99e-f857cface646","Type":"ContainerDied","Data":"c5e9432023600931dcfc67f8bed425b7f7e40849299d5b85c95eae1334fc122b"}
Oct 30 19:07:52.337279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:52.336816    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"
Oct 30 19:07:52.337279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:52.336840    4267 scope.go:122] "RemoveContainer" containerID="10255b45112ff628b7ad5ebd0e20702e9ba5b8a8e783fcdd5a863361e1cb05c1"
Oct 30 19:07:52.600293 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:52.600155    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"]
Oct 30 19:07:52.601529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:52.601475    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4c249f-5494fdc667-mkq4r"]
Oct 30 19:07:53.244940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:07:53.244904    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a56c08-711a-43c4-b99e-f857cface646" path="/var/lib/kubelet/pods/01a56c08-711a-43c4-b99e-f857cface646/volumes"
Oct 30 19:08:02.148252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.148211    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"]
Oct 30 19:08:02.148980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.148960    4267 cpu_manager.go:414] "containerMap: removing container" podUID="01a56c08-711a-43c4-b99e-f857cface646" containerName="payload-pre-processing"
Oct 30 19:08:02.148980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.148978    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a56c08-711a-43c4-b99e-f857cface646" containerName="payload-pre-processing"
Oct 30 19:08:02.149141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.149122    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a56c08-711a-43c4-b99e-f857cface646" containerName="payload-pre-processing"
Oct 30 19:08:02.154411 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.154390    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.190273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.190241    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.190273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.190282    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.190597 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.190387    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1650ffd0-6942-4650-8795-2de75a4fbc87-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.190597 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.190456    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29hdz\" (UniqueName: \"kubernetes.io/projected/1650ffd0-6942-4650-8795-2de75a4fbc87-kube-api-access-29hdz\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.190712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.190622    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.190712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.190676    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.190794 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.190762    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.190850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.190830    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.190957 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.190944    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.292731 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.292690    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.292906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.292811    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.292906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.292861    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.292906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.292895    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.293029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.292953    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1650ffd0-6942-4650-8795-2de75a4fbc87-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.293029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.293019    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29hdz\" (UniqueName: \"kubernetes.io/projected/1650ffd0-6942-4650-8795-2de75a4fbc87-kube-api-access-29hdz\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.293180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.293142    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.293255 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.293181    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.293255 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.293206    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.293371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.293267    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.293730 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.293704    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.293836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.293743    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.293836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.293763    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1650ffd0-6942-4650-8795-2de75a4fbc87-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.293914 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.293890    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.295697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.295668    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.295820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.295762    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.352022 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.351957    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1650ffd0-6942-4650-8795-2de75a4fbc87-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.368928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.368890    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"]
Oct 30 19:08:02.371992 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.371927    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29hdz\" (UniqueName: \"kubernetes.io/projected/1650ffd0-6942-4650-8795-2de75a4fbc87-kube-api-access-29hdz\") pod \"maas-default-gateway-openshift-default-6db586dd6c-m6v84\" (UID: \"1650ffd0-6942-4650-8795-2de75a4fbc87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.471037 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.470997    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:02.919851 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.919751    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"]
Oct 30 19:08:02.954283 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:08:02.954241    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1650ffd0_6942_4650_8795_2de75a4fbc87.slice/crio-05357fc56942089f19bcd2a51de026732f49835bf688deb0ed3d360e05f93bd5 WatchSource:0}: Error finding container 05357fc56942089f19bcd2a51de026732f49835bf688deb0ed3d360e05f93bd5: Status 404 returned error can't find the container with id 05357fc56942089f19bcd2a51de026732f49835bf688deb0ed3d360e05f93bd5
Oct 30 19:08:02.957280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.957237    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:08:02.957409 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.957313    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:08:02.957409 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:02.957342    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:08:03.439042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:03.438992    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" event={"ID":"1650ffd0-6942-4650-8795-2de75a4fbc87","Type":"ContainerStarted","Data":"1322f8488e297236ccc16c629d3ea275f81d41d49414df59470847cadb6193d5"}
Oct 30 19:08:03.439629 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:03.439111    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" event={"ID":"1650ffd0-6942-4650-8795-2de75a4fbc87","Type":"ContainerStarted","Data":"05357fc56942089f19bcd2a51de026732f49835bf688deb0ed3d360e05f93bd5"}
Oct 30 19:08:03.471240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:03.471204    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:03.650450 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:03.650390    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" podStartSLOduration=1.6503699379999999 podStartE2EDuration="1.650369938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:03.607476967 +0000 UTC m=+1774.495394059" watchObservedRunningTime="2026-10-30 19:08:03.650369938 +0000 UTC m=+1774.538287019"
Oct 30 19:08:04.473435 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:04.473405    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6v84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:08:04.473856 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:04.473468    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" podUID="1650ffd0-6942-4650-8795-2de75a4fbc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 19:08:05.471346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:05.471304    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6v84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:08:05.471572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:05.471381    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" podUID="1650ffd0-6942-4650-8795-2de75a4fbc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 19:08:06.471485 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:06.471454    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6v84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:08:06.471919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:06.471517    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" podUID="1650ffd0-6942-4650-8795-2de75a4fbc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 19:08:07.472159 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:07.472109    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6v84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:08:07.472680 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:07.472186    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" podUID="1650ffd0-6942-4650-8795-2de75a4fbc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 19:08:08.471903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:08.471869    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6v84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:08:08.472645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:08.472162    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" podUID="1650ffd0-6942-4650-8795-2de75a4fbc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 19:08:09.474520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:09.473901    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6v84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:08:09.474520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:09.473966    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" podUID="1650ffd0-6942-4650-8795-2de75a4fbc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 19:08:10.471842 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:10.471800    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6v84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:08:10.472038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:10.471903    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" podUID="1650ffd0-6942-4650-8795-2de75a4fbc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 19:08:11.472565 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:11.472508    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-m6v84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:08:11.472565 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:11.472562    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84" podUID="1650ffd0-6942-4650-8795-2de75a4fbc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 19:08:12.816971 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:12.816929    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:12.817481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:12.817406    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:12.818343 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:12.818315    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-m6v84"
Oct 30 19:08:20.159190 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:08:20.159143    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:08:22.416555 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:22.416520    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"]
Oct 30 19:08:22.417464 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:22.416781    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy" containerID="cri-o://e3315d47eeffd515b5e3973693d7669ac05908bce54cbf6f13519ef507e22753" gracePeriod=30
Oct 30 19:08:22.466295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:22.466251    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"]
Oct 30 19:08:22.466572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:22.466537    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy" containerID="cri-o://e9eef03fed5643b81b835a76483bdb1a5e2114d6e925da827b6c2dd07a1578ed" gracePeriod=30
Oct 30 19:08:22.471440 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:22.471402    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"]
Oct 30 19:08:22.471690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:22.471666    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy" containerID="cri-o://0cece4f63bb67fa2e815ef7c5ed7a131215b6891cfa93259e35e9d2ed6d5d805" gracePeriod=30
Oct 30 19:08:22.516926 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:22.516890    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"]
Oct 30 19:08:22.517173 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:22.517145    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy" containerID="cri-o://d7bbcf398118f6b0c8782dc04941e9c67ea3ef486edc81e7b40cd7ab1943de36" gracePeriod=30
Oct 30 19:08:22.899279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:22.899237    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:08:22.899517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:22.899308    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:08:24.055982 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:24.055944    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"]
Oct 30 19:08:24.104506 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:24.104444    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dddb3eca-z465p"]
Oct 30 19:08:25.223108 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:25.223074    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c4e4c5-6ecd-4354-b636-e818b4c0e793" path="/var/lib/kubelet/pods/e3c4e4c5-6ecd-4354-b636-e818b4c0e793/volumes"
Oct 30 19:08:26.173713 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:26.173676    4267 patch_prober.go:29] interesting pod/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:08:26.173903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:26.173737    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:08:27.697948 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.697907    4267 generic.go:356] "Generic (PLEG): container finished" podID="eae2f972-98d9-449c-9934-159b2b027f12" containerID="d7bbcf398118f6b0c8782dc04941e9c67ea3ef486edc81e7b40cd7ab1943de36" exitCode=0
Oct 30 19:08:27.698447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.698044    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" event={"ID":"eae2f972-98d9-449c-9934-159b2b027f12","Type":"ContainerDied","Data":"d7bbcf398118f6b0c8782dc04941e9c67ea3ef486edc81e7b40cd7ab1943de36"}
Oct 30 19:08:27.700619 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.700377    4267 generic.go:356] "Generic (PLEG): container finished" podID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerID="0cece4f63bb67fa2e815ef7c5ed7a131215b6891cfa93259e35e9d2ed6d5d805" exitCode=0
Oct 30 19:08:27.700619 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.700539    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" event={"ID":"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36","Type":"ContainerDied","Data":"0cece4f63bb67fa2e815ef7c5ed7a131215b6891cfa93259e35e9d2ed6d5d805"}
Oct 30 19:08:27.703253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.702972    4267 generic.go:356] "Generic (PLEG): container finished" podID="6981301a-339d-4827-b951-cf81a8e00d2f" containerID="e9eef03fed5643b81b835a76483bdb1a5e2114d6e925da827b6c2dd07a1578ed" exitCode=0
Oct 30 19:08:27.703253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.703174    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" event={"ID":"6981301a-339d-4827-b951-cf81a8e00d2f","Type":"ContainerDied","Data":"e9eef03fed5643b81b835a76483bdb1a5e2114d6e925da827b6c2dd07a1578ed"}
Oct 30 19:08:27.705776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.705655    4267 generic.go:356] "Generic (PLEG): container finished" podID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerID="e3315d47eeffd515b5e3973693d7669ac05908bce54cbf6f13519ef507e22753" exitCode=0
Oct 30 19:08:27.705776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.705703    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" event={"ID":"bcbdd267-f2ff-41fa-9394-4342eef03d4f","Type":"ContainerDied","Data":"e3315d47eeffd515b5e3973693d7669ac05908bce54cbf6f13519ef507e22753"}
Oct 30 19:08:27.810747 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.810709    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 19:08:27.896740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.896640    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.931044    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-podinfo\") pod \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") "
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.931099    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-kube-api-access-bkfzs\" (UniqueName: \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-kube-api-access-bkfzs\") pod \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") "
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.931145    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-envoy\") pod \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") "
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.931200    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istiod-ca-cert\") pod \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") "
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.931376    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-certs\") pod \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") "
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.931432    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-token\" (UniqueName: \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-token\") pod \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") "
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.931615    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-credential-socket\") pod \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") "
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.931649    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-data\") pod \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") "
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.931766    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-socket\") pod \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\" (UID: \"bcbdd267-f2ff-41fa-9394-4342eef03d4f\") "
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.933169    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istiod-ca-cert" pod "bcbdd267-f2ff-41fa-9394-4342eef03d4f" (UID: "bcbdd267-f2ff-41fa-9394-4342eef03d4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.933648    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-socket" pod "bcbdd267-f2ff-41fa-9394-4342eef03d4f" (UID: "bcbdd267-f2ff-41fa-9394-4342eef03d4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.934111    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-certs" pod "bcbdd267-f2ff-41fa-9394-4342eef03d4f" (UID: "bcbdd267-f2ff-41fa-9394-4342eef03d4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.934181    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-data" pod "bcbdd267-f2ff-41fa-9394-4342eef03d4f" (UID: "bcbdd267-f2ff-41fa-9394-4342eef03d4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:27.936544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.934458    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-credential-socket" pod "bcbdd267-f2ff-41fa-9394-4342eef03d4f" (UID: "bcbdd267-f2ff-41fa-9394-4342eef03d4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:27.961061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.960960    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-podinfo" pod "bcbdd267-f2ff-41fa-9394-4342eef03d4f" (UID: "bcbdd267-f2ff-41fa-9394-4342eef03d4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:27.961061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.960984    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-kube-api-access-bkfzs" pod "bcbdd267-f2ff-41fa-9394-4342eef03d4f" (UID: "bcbdd267-f2ff-41fa-9394-4342eef03d4f"). InnerVolumeSpecName "kube-api-access-bkfzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:27.961061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.960946    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-envoy" pod "bcbdd267-f2ff-41fa-9394-4342eef03d4f" (UID: "bcbdd267-f2ff-41fa-9394-4342eef03d4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:27.962636 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.962611    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-token" pod "bcbdd267-f2ff-41fa-9394-4342eef03d4f" (UID: "bcbdd267-f2ff-41fa-9394-4342eef03d4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:27.977357 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.977331    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 19:08:27.981539 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:27.981472    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:08:28.033878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.033825    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-podinfo\") pod \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") "
Oct 30 19:08:28.034165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.033927    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-data\") pod \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") "
Oct 30 19:08:28.034165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.034098    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-credential-socket\") pod \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") "
Oct 30 19:08:28.034165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.034132    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-certs\") pod \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") "
Oct 30 19:08:28.034389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.034186    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-envoy\") pod \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") "
Oct 30 19:08:28.034389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.034219    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-token\" (UniqueName: \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-token\") pod \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") "
Oct 30 19:08:28.034389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.034277    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istiod-ca-cert\") pod \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") "
Oct 30 19:08:28.034631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.034535    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-socket\") pod \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") "
Oct 30 19:08:28.034631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.034584    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-kube-api-access-mpvlc\" (UniqueName: \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-kube-api-access-mpvlc\") pod \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\" (UID: \"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36\") "
Oct 30 19:08:28.035136 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.034802    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-credential-socket" pod "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" (UID: "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.035438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.035403    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.035637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.035618    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.036027 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.036008    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.036220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.036203    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.036349 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.036334    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.036520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.036479    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.036650 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.036636    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.039070 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.039045    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.039186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.035933    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-certs" pod "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" (UID: "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.039186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.036243    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istiod-ca-cert" pod "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" (UID: "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:28.039186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.036289    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-data" pod "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" (UID: "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.039186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.036568    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-socket" pod "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" (UID: "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.039186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.039135    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkfzs\" (UniqueName: \"kubernetes.io/projected/bcbdd267-f2ff-41fa-9394-4342eef03d4f-kube-api-access-bkfzs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.039186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.039150    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcbdd267-f2ff-41fa-9394-4342eef03d4f-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.039660 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.039633    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-token" pod "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" (UID: "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.039776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.039757    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-podinfo" pod "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" (UID: "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:28.040797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.040771    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-kube-api-access-mpvlc" pod "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" (UID: "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36"). InnerVolumeSpecName "kube-api-access-mpvlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.041706 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.041669    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-envoy" pod "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" (UID: "ae1e43fc-08d6-4ea7-8065-14a0d0b77c36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.140709 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.140664    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-envoy\") pod \"6981301a-339d-4827-b951-cf81a8e00d2f\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") "
Oct 30 19:08:28.140924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.140736    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-istio-token\" (UniqueName: \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-istio-token\") pod \"eae2f972-98d9-449c-9934-159b2b027f12\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") "
Oct 30 19:08:28.140924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.140813    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-envoy\") pod \"eae2f972-98d9-449c-9934-159b2b027f12\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") "
Oct 30 19:08:28.140924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.140846    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-socket\") pod \"6981301a-339d-4827-b951-cf81a8e00d2f\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") "
Oct 30 19:08:28.141096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.140941    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-data\") pod \"6981301a-339d-4827-b951-cf81a8e00d2f\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") "
Oct 30 19:08:28.141096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.140973    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6981301a-339d-4827-b951-cf81a8e00d2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6981301a-339d-4827-b951-cf81a8e00d2f-istio-podinfo\") pod \"6981301a-339d-4827-b951-cf81a8e00d2f\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") "
Oct 30 19:08:28.141096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.141021    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eae2f972-98d9-449c-9934-159b2b027f12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae2f972-98d9-449c-9934-159b2b027f12-istio-podinfo\") pod \"eae2f972-98d9-449c-9934-159b2b027f12\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") "
Oct 30 19:08:28.141096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.141068    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-credential-socket\") pod \"eae2f972-98d9-449c-9934-159b2b027f12\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") "
Oct 30 19:08:28.141322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.141103    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-kube-api-access-t7qtw\" (UniqueName: \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-kube-api-access-t7qtw\") pod \"6981301a-339d-4827-b951-cf81a8e00d2f\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") "
Oct 30 19:08:28.141322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.141145    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-certs\") pod \"eae2f972-98d9-449c-9934-159b2b027f12\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") "
Oct 30 19:08:28.141322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.141212    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-socket\") pod \"eae2f972-98d9-449c-9934-159b2b027f12\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") "
Oct 30 19:08:28.141322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.141313    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-istio-token\" (UniqueName: \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-istio-token\") pod \"6981301a-339d-4827-b951-cf81a8e00d2f\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") "
Oct 30 19:08:28.141546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.141502    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-credential-socket" pod "eae2f972-98d9-449c-9934-159b2b027f12" (UID: "eae2f972-98d9-449c-9934-159b2b027f12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.141825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.141796    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-socket" pod "6981301a-339d-4827-b951-cf81a8e00d2f" (UID: "6981301a-339d-4827-b951-cf81a8e00d2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.141825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.141811    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-data" pod "6981301a-339d-4827-b951-cf81a8e00d2f" (UID: "6981301a-339d-4827-b951-cf81a8e00d2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.142081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.142059    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-certs" pod "eae2f972-98d9-449c-9934-159b2b027f12" (UID: "eae2f972-98d9-449c-9934-159b2b027f12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.142354 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.142330    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-socket" pod "eae2f972-98d9-449c-9934-159b2b027f12" (UID: "eae2f972-98d9-449c-9934-159b2b027f12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.142677 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.142654    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-certs\") pod \"6981301a-339d-4827-b951-cf81a8e00d2f\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") "
Oct 30 19:08:28.142767 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.142712    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae2f972-98d9-449c-9934-159b2b027f12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae2f972-98d9-449c-9934-159b2b027f12-istiod-ca-cert\") pod \"eae2f972-98d9-449c-9934-159b2b027f12\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") "
Oct 30 19:08:28.142830 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.142778    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6981301a-339d-4827-b951-cf81a8e00d2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6981301a-339d-4827-b951-cf81a8e00d2f-istiod-ca-cert\") pod \"6981301a-339d-4827-b951-cf81a8e00d2f\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") "
Oct 30 19:08:28.142942 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.142926    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-data\") pod \"eae2f972-98d9-449c-9934-159b2b027f12\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") "
Oct 30 19:08:28.143004 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.142983    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-credential-socket\") pod \"6981301a-339d-4827-b951-cf81a8e00d2f\" (UID: \"6981301a-339d-4827-b951-cf81a8e00d2f\") "
Oct 30 19:08:28.143061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.143041    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-kube-api-access-ghbb8\" (UniqueName: \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-kube-api-access-ghbb8\") pod \"eae2f972-98d9-449c-9934-159b2b027f12\" (UID: \"eae2f972-98d9-449c-9934-159b2b027f12\") "
Oct 30 19:08:28.144037 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144009    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144041    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144057    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144072    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpvlc\" (UniqueName: \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-kube-api-access-mpvlc\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144087    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144101    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144115    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144129    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144506 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144144    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144506 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144158    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144506 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144171    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144506 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144185    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144506 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144199    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.144506 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144233    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-istio-token" pod "eae2f972-98d9-449c-9934-159b2b027f12" (UID: "eae2f972-98d9-449c-9934-159b2b027f12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.144874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144581    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-certs" pod "6981301a-339d-4827-b951-cf81a8e00d2f" (UID: "6981301a-339d-4827-b951-cf81a8e00d2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.144874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144626    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6981301a-339d-4827-b951-cf81a8e00d2f-istiod-ca-cert" pod "6981301a-339d-4827-b951-cf81a8e00d2f" (UID: "6981301a-339d-4827-b951-cf81a8e00d2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:28.144874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144793    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-data" pod "eae2f972-98d9-449c-9934-159b2b027f12" (UID: "eae2f972-98d9-449c-9934-159b2b027f12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.144874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.144853    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eae2f972-98d9-449c-9934-159b2b027f12-istio-podinfo" pod "eae2f972-98d9-449c-9934-159b2b027f12" (UID: "eae2f972-98d9-449c-9934-159b2b027f12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:28.145080 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.145018    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae2f972-98d9-449c-9934-159b2b027f12-istiod-ca-cert" pod "eae2f972-98d9-449c-9934-159b2b027f12" (UID: "eae2f972-98d9-449c-9934-159b2b027f12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:28.145176 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.145152    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-credential-socket" pod "6981301a-339d-4827-b951-cf81a8e00d2f" (UID: "6981301a-339d-4827-b951-cf81a8e00d2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.145823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.145797    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-envoy" pod "6981301a-339d-4827-b951-cf81a8e00d2f" (UID: "6981301a-339d-4827-b951-cf81a8e00d2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.145823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.145810    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6981301a-339d-4827-b951-cf81a8e00d2f-istio-podinfo" pod "6981301a-339d-4827-b951-cf81a8e00d2f" (UID: "6981301a-339d-4827-b951-cf81a8e00d2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:28.146077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.146061    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-envoy" pod "eae2f972-98d9-449c-9934-159b2b027f12" (UID: "eae2f972-98d9-449c-9934-159b2b027f12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:28.146668 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.146632    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-kube-api-access-t7qtw" pod "6981301a-339d-4827-b951-cf81a8e00d2f" (UID: "6981301a-339d-4827-b951-cf81a8e00d2f"). InnerVolumeSpecName "kube-api-access-t7qtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.147621 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.147600    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-istio-token" pod "6981301a-339d-4827-b951-cf81a8e00d2f" (UID: "6981301a-339d-4827-b951-cf81a8e00d2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.148013 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.147992    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-kube-api-access-ghbb8" pod "eae2f972-98d9-449c-9934-159b2b027f12" (UID: "eae2f972-98d9-449c-9934-159b2b027f12"). InnerVolumeSpecName "kube-api-access-ghbb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:28.249729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.249627    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.249729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.249679    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6981301a-339d-4827-b951-cf81a8e00d2f-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.249729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.249695    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae2f972-98d9-449c-9934-159b2b027f12-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.249729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.249710    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7qtw\" (UniqueName: \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-kube-api-access-t7qtw\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.250083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.249725    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6981301a-339d-4827-b951-cf81a8e00d2f-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.250083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.249754    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.250083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.249770    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae2f972-98d9-449c-9934-159b2b027f12-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.250083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.249785    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6981301a-339d-4827-b951-cf81a8e00d2f-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.250083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.249820    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae2f972-98d9-449c-9934-159b2b027f12-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.250083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.250015    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.250083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.250033    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghbb8\" (UniqueName: \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-kube-api-access-ghbb8\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.250083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.250048    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6981301a-339d-4827-b951-cf81a8e00d2f-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.250083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.250064    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae2f972-98d9-449c-9934-159b2b027f12-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:28.718117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.718084    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"
Oct 30 19:08:28.718117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.718100    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp" event={"ID":"bcbdd267-f2ff-41fa-9394-4342eef03d4f","Type":"ContainerDied","Data":"f294ef536edffc5fb803db47f8578d6f39cdbd2af2cf9880089bc3b5088fa4e0"}
Oct 30 19:08:28.718765 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.718164    4267 scope.go:122] "RemoveContainer" containerID="e3315d47eeffd515b5e3973693d7669ac05908bce54cbf6f13519ef507e22753"
Oct 30 19:08:28.720403 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.720371    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq" event={"ID":"eae2f972-98d9-449c-9934-159b2b027f12","Type":"ContainerDied","Data":"2ea9241b7d141e51fddbea11e9617519796342d54d836e4d5a69a3da54d646ef"}
Oct 30 19:08:28.720560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.720381    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"
Oct 30 19:08:28.735641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.735619    4267 scope.go:122] "RemoveContainer" containerID="d7bbcf398118f6b0c8782dc04941e9c67ea3ef486edc81e7b40cd7ab1943de36"
Oct 30 19:08:28.737160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.737122    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt" event={"ID":"ae1e43fc-08d6-4ea7-8065-14a0d0b77c36","Type":"ContainerDied","Data":"aba7f506fb76b9a377f0495aa38023880f13dede9c613f51d9c45219ccc42254"}
Oct 30 19:08:28.737275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.737246    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"
Oct 30 19:08:28.742126 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.742035    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"
Oct 30 19:08:28.742126 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.742071    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2" event={"ID":"6981301a-339d-4827-b951-cf81a8e00d2f","Type":"ContainerDied","Data":"64bedd35e1cb8b1dd82684afb211e62dc87ddd324fabc0884660f0ea583c2703"}
Oct 30 19:08:28.752259 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.752232    4267 scope.go:122] "RemoveContainer" containerID="0cece4f63bb67fa2e815ef7c5ed7a131215b6891cfa93259e35e9d2ed6d5d805"
Oct 30 19:08:28.771690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.766087    4267 scope.go:122] "RemoveContainer" containerID="e9eef03fed5643b81b835a76483bdb1a5e2114d6e925da827b6c2dd07a1578ed"
Oct 30 19:08:28.897866 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.897827    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"]
Oct 30 19:08:28.902413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:28.902382    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xgqqp"]
Oct 30 19:08:29.007469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:29.007375    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"]
Oct 30 19:08:29.021841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:29.021801    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-xnpb2"]
Oct 30 19:08:29.130602 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:29.130560    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"]
Oct 30 19:08:29.135011 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:29.134984    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-554rt"]
Oct 30 19:08:29.224053 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:29.224023    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" path="/var/lib/kubelet/pods/6981301a-339d-4827-b951-cf81a8e00d2f/volumes"
Oct 30 19:08:29.224622 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:29.224607    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" path="/var/lib/kubelet/pods/ae1e43fc-08d6-4ea7-8065-14a0d0b77c36/volumes"
Oct 30 19:08:29.225071 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:29.225057    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" path="/var/lib/kubelet/pods/bcbdd267-f2ff-41fa-9394-4342eef03d4f/volumes"
Oct 30 19:08:29.243949 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:29.243915    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"]
Oct 30 19:08:29.250289 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:29.250258    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-41db7c-openshift-default-b679f98c9-tp5dq"]
Oct 30 19:08:30.682849 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:30.682814    4267 scope.go:122] "RemoveContainer" containerID="5e7be18a1580595983eb1f5988cbf887f35702c6f2980331a11cdec15d391a5e"
Oct 30 19:08:30.693895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:30.693871    4267 scope.go:122] "RemoveContainer" containerID="06c54dd55bea8776ae0c8d761cb3692e30df85fd448136d7bdbd8a567fdfb4b5"
Oct 30 19:08:30.703519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:30.703477    4267 scope.go:122] "RemoveContainer" containerID="d87563cab01a898ac05ad2d94d02133598b79c24a197d9cc6395e65d3f9051d3"
Oct 30 19:08:30.713608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:30.713590    4267 scope.go:122] "RemoveContainer" containerID="07efb2bfd30269da943212fb7c42539b444566b9ce1f4d80c1c72245e17be512"
Oct 30 19:08:31.221699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:31.221666    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae2f972-98d9-449c-9934-159b2b027f12" path="/var/lib/kubelet/pods/eae2f972-98d9-449c-9934-159b2b027f12/volumes"
Oct 30 19:08:33.716519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:33.711938    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"]
Oct 30 19:08:33.716519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:33.712244    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s" podUID="39e14a47-d527-4fc9-b376-ada263d9d5eb" containerName="maas-api" containerID="cri-o://c84c90b2b9b864440194a9548498a6d12df4a436fb790eb24d28c486911137ef" gracePeriod=30
Oct 30 19:08:33.990377 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:33.982926    4267 generic.go:356] "Generic (PLEG): container finished" podID="39e14a47-d527-4fc9-b376-ada263d9d5eb" containerID="c84c90b2b9b864440194a9548498a6d12df4a436fb790eb24d28c486911137ef" exitCode=0
Oct 30 19:08:33.990377 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:33.983142    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s" event={"ID":"39e14a47-d527-4fc9-b376-ada263d9d5eb","Type":"ContainerDied","Data":"c84c90b2b9b864440194a9548498a6d12df4a436fb790eb24d28c486911137ef"}
Oct 30 19:08:34.056962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.056201    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"]
Oct 30 19:08:34.061887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.061849    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d55ee1d-wwpcb"]
Oct 30 19:08:34.227651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.226607    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:08:34.276865 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.276827    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-metrics-certs\") pod \"39e14a47-d527-4fc9-b376-ada263d9d5eb\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") "
Oct 30 19:08:34.277667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.277638    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e14a47-d527-4fc9-b376-ada263d9d5eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e14a47-d527-4fc9-b376-ada263d9d5eb-openshift-service-ca\") pod \"39e14a47-d527-4fc9-b376-ada263d9d5eb\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") "
Oct 30 19:08:34.278117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.278090    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls\") pod \"39e14a47-d527-4fc9-b376-ada263d9d5eb\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") "
Oct 30 19:08:34.278261 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.278204    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e14a47-d527-4fc9-b376-ada263d9d5eb-openshift-service-ca" pod "39e14a47-d527-4fc9-b376-ada263d9d5eb" (UID: "39e14a47-d527-4fc9-b376-ada263d9d5eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:34.278423 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.278392    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e14a47-d527-4fc9-b376-ada263d9d5eb-kube-api-access-pbcf7\" (UniqueName: \"kubernetes.io/projected/39e14a47-d527-4fc9-b376-ada263d9d5eb-kube-api-access-pbcf7\") pod \"39e14a47-d527-4fc9-b376-ada263d9d5eb\" (UID: \"39e14a47-d527-4fc9-b376-ada263d9d5eb\") "
Oct 30 19:08:34.279356 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.279333    4267 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e14a47-d527-4fc9-b376-ada263d9d5eb-openshift-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:34.286353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.285766    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e14a47-d527-4fc9-b376-ada263d9d5eb-kube-api-access-pbcf7" pod "39e14a47-d527-4fc9-b376-ada263d9d5eb" (UID: "39e14a47-d527-4fc9-b376-ada263d9d5eb"). InnerVolumeSpecName "kube-api-access-pbcf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:34.290305 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.289910    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-metrics-certs" pod "39e14a47-d527-4fc9-b376-ada263d9d5eb" (UID: "39e14a47-d527-4fc9-b376-ada263d9d5eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:08:34.291472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.291445    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls" pod "39e14a47-d527-4fc9-b376-ada263d9d5eb" (UID: "39e14a47-d527-4fc9-b376-ada263d9d5eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:08:34.381625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.381564    4267 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-metrics-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:34.381625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.381629    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39e14a47-d527-4fc9-b376-ada263d9d5eb-maas-api-tls\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:34.381889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.381646    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbcf7\" (UniqueName: \"kubernetes.io/projected/39e14a47-d527-4fc9-b376-ada263d9d5eb-kube-api-access-pbcf7\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:08:34.994405 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.994361    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s" event={"ID":"39e14a47-d527-4fc9-b376-ada263d9d5eb","Type":"ContainerDied","Data":"aa2cb5f553df6792af7fb5c179cbdaf6b1303547c115245fe2bb9cba919af1dd"}
Oct 30 19:08:34.995042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.994442    4267 scope.go:122] "RemoveContainer" containerID="c84c90b2b9b864440194a9548498a6d12df4a436fb790eb24d28c486911137ef"
Oct 30 19:08:34.995171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:34.995143    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"
Oct 30 19:08:35.163445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:35.163414    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"]
Oct 30 19:08:35.211802 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:35.211769    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eb4c249f-65cd9f4975-glp7s"]
Oct 30 19:08:35.224368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:35.224335    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259647e9-76ef-4b48-ad76-8d463577981d" path="/var/lib/kubelet/pods/259647e9-76ef-4b48-ad76-8d463577981d/volumes"
Oct 30 19:08:35.224885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:35.224870    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e14a47-d527-4fc9-b376-ada263d9d5eb" path="/var/lib/kubelet/pods/39e14a47-d527-4fc9-b376-ada263d9d5eb/volumes"
Oct 30 19:08:45.001286 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.001243    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"]
Oct 30 19:08:45.002722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.002698    4267 cpu_manager.go:414] "containerMap: removing container" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy"
Oct 30 19:08:45.002823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.002724    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy"
Oct 30 19:08:45.002823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.002737    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy"
Oct 30 19:08:45.002823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.002745    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy"
Oct 30 19:08:45.002823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.002767    4267 cpu_manager.go:414] "containerMap: removing container" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy"
Oct 30 19:08:45.002823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.002775    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy"
Oct 30 19:08:45.002823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.002787    4267 cpu_manager.go:414] "containerMap: removing container" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy"
Oct 30 19:08:45.002823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.002795    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy"
Oct 30 19:08:45.003030 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.002845    4267 cpu_manager.go:414] "containerMap: removing container" podUID="39e14a47-d527-4fc9-b376-ada263d9d5eb" containerName="maas-api"
Oct 30 19:08:45.003030 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.002852    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e14a47-d527-4fc9-b376-ada263d9d5eb" containerName="maas-api"
Oct 30 19:08:45.003030 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.003016    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="6981301a-339d-4827-b951-cf81a8e00d2f" containerName="istio-proxy"
Oct 30 19:08:45.003030 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.003027    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae2f972-98d9-449c-9934-159b2b027f12" containerName="istio-proxy"
Oct 30 19:08:45.003154 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.003045    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e14a47-d527-4fc9-b376-ada263d9d5eb" containerName="maas-api"
Oct 30 19:08:45.003154 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.003062    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbdd267-f2ff-41fa-9394-4342eef03d4f" containerName="istio-proxy"
Oct 30 19:08:45.003154 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.003073    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1e43fc-08d6-4ea7-8065-14a0d0b77c36" containerName="istio-proxy"
Oct 30 19:08:45.011238 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.011207    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:08:45.060568 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.060535    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"]
Oct 30 19:08:45.063783 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.063754    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f26b1d\""
Oct 30 19:08:45.156512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.139700    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add9774b-fbc8-4434-a91b-ed2be5fba187-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7\" (UID: \"add9774b-fbc8-4434-a91b-ed2be5fba187\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:08:45.156512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.140032    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphh5\" (UniqueName: \"kubernetes.io/projected/add9774b-fbc8-4434-a91b-ed2be5fba187-kube-api-access-hphh5\") pod \"payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7\" (UID: \"add9774b-fbc8-4434-a91b-ed2be5fba187\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:08:45.156512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.156284    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"]
Oct 30 19:08:45.181515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.169923    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:08:45.235547 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.234844    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f26b1d-dockercfg-j6hqr\""
Oct 30 19:08:45.245637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.245597    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add9774b-fbc8-4434-a91b-ed2be5fba187-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7\" (UID: \"add9774b-fbc8-4434-a91b-ed2be5fba187\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:08:45.245810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.245680    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34103503-dc6a-4ca3-8644-e8bb750fc414-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s\" (UID: \"34103503-dc6a-4ca3-8644-e8bb750fc414\") " pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:08:45.245810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.245742    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphh5\" (UniqueName: \"kubernetes.io/projected/add9774b-fbc8-4434-a91b-ed2be5fba187-kube-api-access-hphh5\") pod \"payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7\" (UID: \"add9774b-fbc8-4434-a91b-ed2be5fba187\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:08:45.245810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.245775    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2kf\" (UniqueName: \"kubernetes.io/projected/34103503-dc6a-4ca3-8644-e8bb750fc414-kube-api-access-2q2kf\") pod \"payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s\" (UID: \"34103503-dc6a-4ca3-8644-e8bb750fc414\") " pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:08:45.247071 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.247042    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add9774b-fbc8-4434-a91b-ed2be5fba187-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7\" (UID: \"add9774b-fbc8-4434-a91b-ed2be5fba187\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:08:45.249050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.249027    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"]
Oct 30 19:08:45.318549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.318460    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphh5\" (UniqueName: \"kubernetes.io/projected/add9774b-fbc8-4434-a91b-ed2be5fba187-kube-api-access-hphh5\") pod \"payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7\" (UID: \"add9774b-fbc8-4434-a91b-ed2be5fba187\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:08:45.333924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.333889    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:08:45.349260 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.348076    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34103503-dc6a-4ca3-8644-e8bb750fc414-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s\" (UID: \"34103503-dc6a-4ca3-8644-e8bb750fc414\") " pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:08:45.349260 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.348167    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2kf\" (UniqueName: \"kubernetes.io/projected/34103503-dc6a-4ca3-8644-e8bb750fc414-kube-api-access-2q2kf\") pod \"payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s\" (UID: \"34103503-dc6a-4ca3-8644-e8bb750fc414\") " pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:08:45.353511 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.349962    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34103503-dc6a-4ca3-8644-e8bb750fc414-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s\" (UID: \"34103503-dc6a-4ca3-8644-e8bb750fc414\") " pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:08:45.418209 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.418162    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2kf\" (UniqueName: \"kubernetes.io/projected/34103503-dc6a-4ca3-8644-e8bb750fc414-kube-api-access-2q2kf\") pod \"payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s\" (UID: \"34103503-dc6a-4ca3-8644-e8bb750fc414\") " pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:08:45.540941 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.539398    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:08:45.774820 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:08:45.774789    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd9774b_fbc8_4434_a91b_ed2be5fba187.slice/crio-80445e402ce9dc3a1b416f9fafc1b4df6b1d7c359b1455d75fd4f7d167aca7fc WatchSource:0}: Error finding container 80445e402ce9dc3a1b416f9fafc1b4df6b1d7c359b1455d75fd4f7d167aca7fc: Status 404 returned error can't find the container with id 80445e402ce9dc3a1b416f9fafc1b4df6b1d7c359b1455d75fd4f7d167aca7fc
Oct 30 19:08:45.775300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.775275    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"]
Oct 30 19:08:45.933837 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:45.933800    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"]
Oct 30 19:08:45.934262 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:08:45.934238    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34103503_dc6a_4ca3_8644_e8bb750fc414.slice/crio-d732ec1a38dbd02e0897f47e52f57c93247b25ebd24aae28e2c15052dc7ac8aa WatchSource:0}: Error finding container d732ec1a38dbd02e0897f47e52f57c93247b25ebd24aae28e2c15052dc7ac8aa: Status 404 returned error can't find the container with id d732ec1a38dbd02e0897f47e52f57c93247b25ebd24aae28e2c15052dc7ac8aa
Oct 30 19:08:46.114234 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:46.114195    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7" event={"ID":"add9774b-fbc8-4434-a91b-ed2be5fba187","Type":"ContainerStarted","Data":"e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad"}
Oct 30 19:08:46.114655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:46.114270    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:08:46.114655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:46.114294    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7" event={"ID":"add9774b-fbc8-4434-a91b-ed2be5fba187","Type":"ContainerStarted","Data":"80445e402ce9dc3a1b416f9fafc1b4df6b1d7c359b1455d75fd4f7d167aca7fc"}
Oct 30 19:08:46.116071 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:46.116041    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s" event={"ID":"34103503-dc6a-4ca3-8644-e8bb750fc414","Type":"ContainerStarted","Data":"2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46"}
Oct 30 19:08:46.116189 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:46.116087    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s" event={"ID":"34103503-dc6a-4ca3-8644-e8bb750fc414","Type":"ContainerStarted","Data":"d732ec1a38dbd02e0897f47e52f57c93247b25ebd24aae28e2c15052dc7ac8aa"}
Oct 30 19:08:46.116189 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:46.116142    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:08:46.232337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:46.232274    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7" podStartSLOduration=2.232254107 podStartE2EDuration="2.232254107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:46.231614232 +0000 UTC m=+1817.119531322" watchObservedRunningTime="2026-10-30 19:08:46.232254107 +0000 UTC m=+1817.120171197"
Oct 30 19:08:46.372425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:46.370352    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s" podStartSLOduration=1.3703336830000001 podStartE2EDuration="1.370333683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:46.369745662 +0000 UTC m=+1817.257662752" watchObservedRunningTime="2026-10-30 19:08:46.370333683 +0000 UTC m=+1817.258250773"
Oct 30 19:08:50.160986 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:08:50.160948    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:08:51.331183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.331145    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"]
Oct 30 19:08:51.335717 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.335699    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:08:51.405515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.401641    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"]
Oct 30 19:08:51.449515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.443827    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0b8d0f-dockercfg-q855j\""
Oct 30 19:08:51.449515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.443924    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0b8d0f\""
Oct 30 19:08:51.545520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.542872    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2226498d-e1e2-446d-a802-a1b107bd5cf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk\" (UID: \"2226498d-e1e2-446d-a802-a1b107bd5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:08:51.545520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.543036    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwpr\" (UniqueName: \"kubernetes.io/projected/2226498d-e1e2-446d-a802-a1b107bd5cf5-kube-api-access-rmwpr\") pod \"payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk\" (UID: \"2226498d-e1e2-446d-a802-a1b107bd5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:08:51.560591 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.547505    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"]
Oct 30 19:08:51.655559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.645936    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwpr\" (UniqueName: \"kubernetes.io/projected/2226498d-e1e2-446d-a802-a1b107bd5cf5-kube-api-access-rmwpr\") pod \"payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk\" (UID: \"2226498d-e1e2-446d-a802-a1b107bd5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:08:51.655559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.646256    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2226498d-e1e2-446d-a802-a1b107bd5cf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk\" (UID: \"2226498d-e1e2-446d-a802-a1b107bd5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:08:51.655559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.647271    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2226498d-e1e2-446d-a802-a1b107bd5cf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk\" (UID: \"2226498d-e1e2-446d-a802-a1b107bd5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:08:51.655559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.647709    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:08:51.672885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.669790    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"]
Oct 30 19:08:51.706103 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.706067    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwpr\" (UniqueName: \"kubernetes.io/projected/2226498d-e1e2-446d-a802-a1b107bd5cf5-kube-api-access-rmwpr\") pod \"payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk\" (UID: \"2226498d-e1e2-446d-a802-a1b107bd5cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:08:51.748050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.748013    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4rkw\" (UniqueName: \"kubernetes.io/projected/93d1eca9-8bcc-42d2-a86e-9719996dbb22-kube-api-access-r4rkw\") pod \"payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck\" (UID: \"93d1eca9-8bcc-42d2-a86e-9719996dbb22\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:08:51.748307 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.748114    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d1eca9-8bcc-42d2-a86e-9719996dbb22-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck\" (UID: \"93d1eca9-8bcc-42d2-a86e-9719996dbb22\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:08:51.849526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.849479    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4rkw\" (UniqueName: \"kubernetes.io/projected/93d1eca9-8bcc-42d2-a86e-9719996dbb22-kube-api-access-r4rkw\") pod \"payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck\" (UID: \"93d1eca9-8bcc-42d2-a86e-9719996dbb22\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:08:51.849740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.849578    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d1eca9-8bcc-42d2-a86e-9719996dbb22-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck\" (UID: \"93d1eca9-8bcc-42d2-a86e-9719996dbb22\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:08:51.850334 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.850311    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d1eca9-8bcc-42d2-a86e-9719996dbb22-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck\" (UID: \"93d1eca9-8bcc-42d2-a86e-9719996dbb22\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:08:51.907147 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.907075    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4rkw\" (UniqueName: \"kubernetes.io/projected/93d1eca9-8bcc-42d2-a86e-9719996dbb22-kube-api-access-r4rkw\") pod \"payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck\" (UID: \"93d1eca9-8bcc-42d2-a86e-9719996dbb22\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:08:51.954308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.954271    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:08:51.970381 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:51.970348    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:08:52.400563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:52.400525    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"]
Oct 30 19:08:52.425442 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:08:52.425410    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2226498d_e1e2_446d_a802_a1b107bd5cf5.slice/crio-27c2daf1440d0cfbdc9695fc2dc1b76e8ef386285ee35fe0fb766b4cdc810693 WatchSource:0}: Error finding container 27c2daf1440d0cfbdc9695fc2dc1b76e8ef386285ee35fe0fb766b4cdc810693: Status 404 returned error can't find the container with id 27c2daf1440d0cfbdc9695fc2dc1b76e8ef386285ee35fe0fb766b4cdc810693
Oct 30 19:08:52.585697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:52.585669    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"]
Oct 30 19:08:52.585804 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:08:52.585778    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d1eca9_8bcc_42d2_a86e_9719996dbb22.slice/crio-194cc35a288d7ca7a856291d57677207bb69d8c039d234df200935690a9aaaaa WatchSource:0}: Error finding container 194cc35a288d7ca7a856291d57677207bb69d8c039d234df200935690a9aaaaa: Status 404 returned error can't find the container with id 194cc35a288d7ca7a856291d57677207bb69d8c039d234df200935690a9aaaaa
Oct 30 19:08:53.359001 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:53.358937    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck" event={"ID":"93d1eca9-8bcc-42d2-a86e-9719996dbb22","Type":"ContainerStarted","Data":"0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc"}
Oct 30 19:08:53.359327 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:53.359033    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck" event={"ID":"93d1eca9-8bcc-42d2-a86e-9719996dbb22","Type":"ContainerStarted","Data":"194cc35a288d7ca7a856291d57677207bb69d8c039d234df200935690a9aaaaa"}
Oct 30 19:08:53.359327 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:53.359182    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:08:53.361451 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:53.361426    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk" event={"ID":"2226498d-e1e2-446d-a802-a1b107bd5cf5","Type":"ContainerStarted","Data":"0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31"}
Oct 30 19:08:53.361597 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:53.361469    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk" event={"ID":"2226498d-e1e2-446d-a802-a1b107bd5cf5","Type":"ContainerStarted","Data":"27c2daf1440d0cfbdc9695fc2dc1b76e8ef386285ee35fe0fb766b4cdc810693"}
Oct 30 19:08:53.361597 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:53.361563    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:08:53.481029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:53.480971    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck" podStartSLOduration=2.480956705 podStartE2EDuration="2.480956705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:53.47734147 +0000 UTC m=+1824.365258560" watchObservedRunningTime="2026-10-30 19:08:53.480956705 +0000 UTC m=+1824.368873833"
Oct 30 19:08:53.614604 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:53.613740    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk" podStartSLOduration=2.613716713 podStartE2EDuration="2.613716713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:53.596679736 +0000 UTC m=+1824.484596826" watchObservedRunningTime="2026-10-30 19:08:53.613716713 +0000 UTC m=+1824.501633808"
Oct 30 19:08:57.129711 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:57.129671    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:08:57.130116 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:08:57.129750    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:09:00.737285 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.737238    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"]
Oct 30 19:09:00.749730 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.749697    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:00.803301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.803264    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"]
Oct 30 19:09:00.847674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.847637    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:09:00.847874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.847701    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:09:00.852537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.852486    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e261a7fd-dthgb\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:00.852704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.852683    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e261a7fd-dthgb\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:00.852851 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.852834    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt26q\" (UniqueName: \"kubernetes.io/projected/22d25088-7507-4245-b9ca-7b9f18a03dc2-kube-api-access-mt26q\") pod \"maas-api-revoke-sub-e261a7fd-dthgb\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:00.954188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.954149    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e261a7fd-dthgb\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:00.954417 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.954346    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt26q\" (UniqueName: \"kubernetes.io/projected/22d25088-7507-4245-b9ca-7b9f18a03dc2-kube-api-access-mt26q\") pod \"maas-api-revoke-sub-e261a7fd-dthgb\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:00.954505 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.954467    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e261a7fd-dthgb\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:00.955133 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.955107    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e261a7fd-dthgb\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:00.956693 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:00.956671    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e261a7fd-dthgb\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:01.011915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:01.011844    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt26q\" (UniqueName: \"kubernetes.io/projected/22d25088-7507-4245-b9ca-7b9f18a03dc2-kube-api-access-mt26q\") pod \"maas-api-revoke-sub-e261a7fd-dthgb\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:01.065620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:01.065584    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:01.456878 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:09:01.456843    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d25088_7507_4245_b9ca_7b9f18a03dc2.slice/crio-31c6f17f5250132a45cf9924e1eca25ff33df01b65c1423d1a8ac783f2517c4c WatchSource:0}: Error finding container 31c6f17f5250132a45cf9924e1eca25ff33df01b65c1423d1a8ac783f2517c4c: Status 404 returned error can't find the container with id 31c6f17f5250132a45cf9924e1eca25ff33df01b65c1423d1a8ac783f2517c4c
Oct 30 19:09:01.457244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:01.457222    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"]
Oct 30 19:09:02.446293 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.446258    4267 generic.go:356] "Generic (PLEG): container finished" podID="22d25088-7507-4245-b9ca-7b9f18a03dc2" containerID="26ac5e2a3fb4146ad781e5d254afcd66915a3027c2cce027aab3d122c61e02bf" exitCode=0
Oct 30 19:09:02.447475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.446330    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb" event={"ID":"22d25088-7507-4245-b9ca-7b9f18a03dc2","Type":"ContainerDied","Data":"26ac5e2a3fb4146ad781e5d254afcd66915a3027c2cce027aab3d122c61e02bf"}
Oct 30 19:09:02.447475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.446386    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb" event={"ID":"22d25088-7507-4245-b9ca-7b9f18a03dc2","Type":"ContainerStarted","Data":"31c6f17f5250132a45cf9924e1eca25ff33df01b65c1423d1a8ac783f2517c4c"}
Oct 30 19:09:02.695036 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.694998    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"]
Oct 30 19:09:02.699699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.699677    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:02.761376 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.761337    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"]
Oct 30 19:09:02.788278 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.788235    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2fkn\" (UniqueName: \"kubernetes.io/projected/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-kube-api-access-q2fkn\") pod \"maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:02.788470 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.788360    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:02.788470 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.788425    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:02.890601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.890563    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2fkn\" (UniqueName: \"kubernetes.io/projected/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-kube-api-access-q2fkn\") pod \"maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:02.890800 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.890647    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:02.890800 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.890674    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:02.891333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.891313    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:02.893160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.893139    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:02.948985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:02.948911    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2fkn\" (UniqueName: \"kubernetes.io/projected/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-kube-api-access-q2fkn\") pod \"maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:03.013974 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.013933    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:03.406207 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:09:03.406176    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8bcdd4_9b3f_4a44_a2e9_af6c4e76d134.slice/crio-0063bd91fb80092f33ea3980cda275eb9508d1c3d19c806a773632f69faf1bdb WatchSource:0}: Error finding container 0063bd91fb80092f33ea3980cda275eb9508d1c3d19c806a773632f69faf1bdb: Status 404 returned error can't find the container with id 0063bd91fb80092f33ea3980cda275eb9508d1c3d19c806a773632f69faf1bdb
Oct 30 19:09:03.406407 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.406380    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"]
Oct 30 19:09:03.455889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.455851    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6" event={"ID":"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134","Type":"ContainerStarted","Data":"0063bd91fb80092f33ea3980cda275eb9508d1c3d19c806a773632f69faf1bdb"}
Oct 30 19:09:03.646906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.646867    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:03.805009 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.804969    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d25088-7507-4245-b9ca-7b9f18a03dc2-kube-api-access-mt26q\" (UniqueName: \"kubernetes.io/projected/22d25088-7507-4245-b9ca-7b9f18a03dc2-kube-api-access-mt26q\") pod \"22d25088-7507-4245-b9ca-7b9f18a03dc2\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") "
Oct 30 19:09:03.805282 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.805244    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-xks-ca\") pod \"22d25088-7507-4245-b9ca-7b9f18a03dc2\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") "
Oct 30 19:09:03.805390 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.805360    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-openshift-ca\") pod \"22d25088-7507-4245-b9ca-7b9f18a03dc2\" (UID: \"22d25088-7507-4245-b9ca-7b9f18a03dc2\") "
Oct 30 19:09:03.806071 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.806046    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-openshift-ca" pod "22d25088-7507-4245-b9ca-7b9f18a03dc2" (UID: "22d25088-7507-4245-b9ca-7b9f18a03dc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:03.809706 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.809679    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-xks-ca" pod "22d25088-7507-4245-b9ca-7b9f18a03dc2" (UID: "22d25088-7507-4245-b9ca-7b9f18a03dc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:09:03.809835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.809719    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d25088-7507-4245-b9ca-7b9f18a03dc2-kube-api-access-mt26q" pod "22d25088-7507-4245-b9ca-7b9f18a03dc2" (UID: "22d25088-7507-4245-b9ca-7b9f18a03dc2"). InnerVolumeSpecName "kube-api-access-mt26q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:03.908221 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.908178    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:03.908221 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.908214    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt26q\" (UniqueName: \"kubernetes.io/projected/22d25088-7507-4245-b9ca-7b9f18a03dc2-kube-api-access-mt26q\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:03.908221 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:03.908226    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d25088-7507-4245-b9ca-7b9f18a03dc2-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:04.372886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:04.372854    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:09:04.373045 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:04.372932    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:09:04.466846 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:04.466809    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6" event={"ID":"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134","Type":"ContainerStarted","Data":"39e73407e39adb2bc3c796f87394bafd9d0c0b32d931ef68ca51f60179a6d40e"}
Oct 30 19:09:04.468314 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:04.468287    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb" event={"ID":"22d25088-7507-4245-b9ca-7b9f18a03dc2","Type":"ContainerDied","Data":"31c6f17f5250132a45cf9924e1eca25ff33df01b65c1423d1a8ac783f2517c4c"}
Oct 30 19:09:04.468438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:04.468335    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31c6f17f5250132a45cf9924e1eca25ff33df01b65c1423d1a8ac783f2517c4c"
Oct 30 19:09:04.468438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:04.468293    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"
Oct 30 19:09:04.725176 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:04.725120    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6" podStartSLOduration=2.725106165 podStartE2EDuration="2.725106165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:04.722718635 +0000 UTC m=+1835.610635725" watchObservedRunningTime="2026-10-30 19:09:04.725106165 +0000 UTC m=+1835.613023253"
Oct 30 19:09:05.477810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:05.477767    4267 generic.go:356] "Generic (PLEG): container finished" podID="0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134" containerID="39e73407e39adb2bc3c796f87394bafd9d0c0b32d931ef68ca51f60179a6d40e" exitCode=0
Oct 30 19:09:05.478281 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:05.477827    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6" event={"ID":"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134","Type":"ContainerDied","Data":"39e73407e39adb2bc3c796f87394bafd9d0c0b32d931ef68ca51f60179a6d40e"}
Oct 30 19:09:06.688564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:06.688529    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:06.769249 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:06.769161    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-openshift-ca\") pod \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") "
Oct 30 19:09:06.769407 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:06.769307    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-xks-ca\") pod \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") "
Oct 30 19:09:06.769573 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:06.769551    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-kube-api-access-q2fkn\" (UniqueName: \"kubernetes.io/projected/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-kube-api-access-q2fkn\") pod \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\" (UID: \"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134\") "
Oct 30 19:09:06.769659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:06.769637    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-openshift-ca" pod "0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134" (UID: "0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:06.770248 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:06.770223    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:06.771532 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:06.771483    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-xks-ca" pod "0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134" (UID: "0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:09:06.771641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:06.771571    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-kube-api-access-q2fkn" pod "0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134" (UID: "0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134"). InnerVolumeSpecName "kube-api-access-q2fkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:06.872788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:06.872744    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:06.872788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:06.872791    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2fkn\" (UniqueName: \"kubernetes.io/projected/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134-kube-api-access-q2fkn\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:07.500285 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:07.500193    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6" event={"ID":"0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134","Type":"ContainerDied","Data":"0063bd91fb80092f33ea3980cda275eb9508d1c3d19c806a773632f69faf1bdb"}
Oct 30 19:09:07.500285 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:07.500222    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"
Oct 30 19:09:07.500285 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:07.500248    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0063bd91fb80092f33ea3980cda275eb9508d1c3d19c806a773632f69faf1bdb"
Oct 30 19:09:09.119292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:09.119254    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"]
Oct 30 19:09:09.120160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:09.120122    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy" containerID="cri-o://e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a" gracePeriod=30
Oct 30 19:09:13.658074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:13.657975    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:13.658074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:13.658056    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:14.267234 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.267201    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"]
Oct 30 19:09:14.390576 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.390543    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"]
Oct 30 19:09:14.433860 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.433831    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:09:14.449903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.449867    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-envoy\") pod \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") "
Oct 30 19:09:14.450103 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.449957    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-credential-socket\") pod \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") "
Oct 30 19:09:14.450103 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450017    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-podinfo\") pod \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") "
Oct 30 19:09:14.450227 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450166    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-certs\") pod \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") "
Oct 30 19:09:14.450227 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450220    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-data\") pod \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") "
Oct 30 19:09:14.450326 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450272    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-credential-socket" pod "09c7ce5b-c4f5-4a31-807f-e2ba352d5302" (UID: "09c7ce5b-c4f5-4a31-807f-e2ba352d5302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:14.450326 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450288    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istiod-ca-cert\") pod \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") "
Oct 30 19:09:14.450467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450444    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-kube-api-access-c9bp6\" (UniqueName: \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-kube-api-access-c9bp6\") pod \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") "
Oct 30 19:09:14.450709 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450689    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-token\" (UniqueName: \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-token\") pod \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") "
Oct 30 19:09:14.450886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450469    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-certs" pod "09c7ce5b-c4f5-4a31-807f-e2ba352d5302" (UID: "09c7ce5b-c4f5-4a31-807f-e2ba352d5302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:14.450971 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450605    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-data" pod "09c7ce5b-c4f5-4a31-807f-e2ba352d5302" (UID: "09c7ce5b-c4f5-4a31-807f-e2ba352d5302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:14.450971 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450692    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istiod-ca-cert" pod "09c7ce5b-c4f5-4a31-807f-e2ba352d5302" (UID: "09c7ce5b-c4f5-4a31-807f-e2ba352d5302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:14.450971 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.450872    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-socket\") pod \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\" (UID: \"09c7ce5b-c4f5-4a31-807f-e2ba352d5302\") "
Oct 30 19:09:14.451204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.451179    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-socket" pod "09c7ce5b-c4f5-4a31-807f-e2ba352d5302" (UID: "09c7ce5b-c4f5-4a31-807f-e2ba352d5302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:14.451684 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.451663    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:14.451775 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.451689    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:14.451775 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.451707    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:14.451775 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.451721    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:14.451775 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.451736    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:14.452607 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.452580    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-envoy" pod "09c7ce5b-c4f5-4a31-807f-e2ba352d5302" (UID: "09c7ce5b-c4f5-4a31-807f-e2ba352d5302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:14.452744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.452718    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-podinfo" pod "09c7ce5b-c4f5-4a31-807f-e2ba352d5302" (UID: "09c7ce5b-c4f5-4a31-807f-e2ba352d5302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:14.452863 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.452845    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-kube-api-access-c9bp6" pod "09c7ce5b-c4f5-4a31-807f-e2ba352d5302" (UID: "09c7ce5b-c4f5-4a31-807f-e2ba352d5302"). InnerVolumeSpecName "kube-api-access-c9bp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:14.453018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.453002    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-token" pod "09c7ce5b-c4f5-4a31-807f-e2ba352d5302" (UID: "09c7ce5b-c4f5-4a31-807f-e2ba352d5302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:14.552862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.552820    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bp6\" (UniqueName: \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-kube-api-access-c9bp6\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:14.552862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.552852    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:14.552862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.552862    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:14.552862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.552871    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c7ce5b-c4f5-4a31-807f-e2ba352d5302-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:14.576914 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.576531    4267 generic.go:356] "Generic (PLEG): container finished" podID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerID="e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a" exitCode=0
Oct 30 19:09:14.576914 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.576570    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"
Oct 30 19:09:14.576914 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.576593    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" event={"ID":"09c7ce5b-c4f5-4a31-807f-e2ba352d5302","Type":"ContainerDied","Data":"e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a"}
Oct 30 19:09:14.576914 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.576692    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65" event={"ID":"09c7ce5b-c4f5-4a31-807f-e2ba352d5302","Type":"ContainerDied","Data":"475b22597918a65c9d3c5fc51120396fd04e8fa0b0461450bdca7d6656e43f30"}
Oct 30 19:09:14.576914 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.576738    4267 scope.go:122] "RemoveContainer" containerID="e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a"
Oct 30 19:09:14.593459 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.593427    4267 scope.go:122] "RemoveContainer" containerID="e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a"
Oct 30 19:09:14.593810 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:09:14.593780    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a\": container with ID starting with e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a not found: ID does not exist" containerID="e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a"
Oct 30 19:09:14.593927 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.593822    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a"} err="failed to get container status \"e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a\": rpc error: code = NotFound desc = could not find container \"e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a\": container with ID starting with e98c63481a870dd63dccb777540ddbb8fb256b6eaa00f650843324e4ea2b0a6a not found: ID does not exist"
Oct 30 19:09:14.750238 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.750196    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"]
Oct 30 19:09:14.755149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:14.755123    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-74b65"]
Oct 30 19:09:15.222813 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:15.222780    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" path="/var/lib/kubelet/pods/09c7ce5b-c4f5-4a31-807f-e2ba352d5302/volumes"
Oct 30 19:09:17.723765 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.723731    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"]
Oct 30 19:09:17.724424 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.724410    4267 cpu_manager.go:414] "containerMap: removing container" podUID="22d25088-7507-4245-b9ca-7b9f18a03dc2" containerName="revoke-keys"
Oct 30 19:09:17.724475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.724425    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d25088-7507-4245-b9ca-7b9f18a03dc2" containerName="revoke-keys"
Oct 30 19:09:17.724475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.724436    4267 cpu_manager.go:414] "containerMap: removing container" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy"
Oct 30 19:09:17.724475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.724441    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy"
Oct 30 19:09:17.724475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.724454    4267 cpu_manager.go:414] "containerMap: removing container" podUID="0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134" containerName="revoke-keys"
Oct 30 19:09:17.724475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.724460    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134" containerName="revoke-keys"
Oct 30 19:09:17.724667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.724601    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d25088-7507-4245-b9ca-7b9f18a03dc2" containerName="revoke-keys"
Oct 30 19:09:17.724667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.724608    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c7ce5b-c4f5-4a31-807f-e2ba352d5302" containerName="istio-proxy"
Oct 30 19:09:17.724667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.724615    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134" containerName="revoke-keys"
Oct 30 19:09:17.729699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.729675    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.809210 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.809165    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e17b05e-8313-497d-a3d2-abfd08e4db51-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.809418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.809225    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.809418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.809297    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.809418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.809359    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52flr\" (UniqueName: \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-kube-api-access-52flr\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.809418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.809391    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.810235 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.809539    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.810235 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.809564    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.810235 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.809761    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.810235 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.809785    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.864018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.863976    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"]
Oct 30 19:09:17.906433 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.906400    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-87ed12-openshift-default-dockercfg-f4qfc\""
Oct 30 19:09:17.910822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.910782    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.910822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.910824    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.910988    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911022    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911117    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e17b05e-8313-497d-a3d2-abfd08e4db51-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911147    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911176    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911208    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52flr\" (UniqueName: \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-kube-api-access-52flr\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911247    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911441 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911248    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911587    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911774 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911750    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.911880 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911854    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.912025 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.911995    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e17b05e-8313-497d-a3d2-abfd08e4db51-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.913783 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.913758    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.913903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.913881    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.970802 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.970734    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:17.970961 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:17.970864    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52flr\" (UniqueName: \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-kube-api-access-52flr\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:18.045381 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:18.045285    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:18.498185 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:09:18.498154    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e17b05e_8313_497d_a3d2_abfd08e4db51.slice/crio-b080f63a8369a63632ad286f588eb9486c496aaa47c7d53c10ff592751568b58 WatchSource:0}: Error finding container b080f63a8369a63632ad286f588eb9486c496aaa47c7d53c10ff592751568b58: Status 404 returned error can't find the container with id b080f63a8369a63632ad286f588eb9486c496aaa47c7d53c10ff592751568b58
Oct 30 19:09:18.498881 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:18.498856    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"]
Oct 30 19:09:18.501052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:18.501011    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:09:18.501171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:18.501099    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:09:18.501171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:18.501149    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:09:18.614695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:18.614656    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" event={"ID":"1e17b05e-8313-497d-a3d2-abfd08e4db51","Type":"ContainerStarted","Data":"92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606"}
Oct 30 19:09:18.614889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:18.614725    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" event={"ID":"1e17b05e-8313-497d-a3d2-abfd08e4db51","Type":"ContainerStarted","Data":"b080f63a8369a63632ad286f588eb9486c496aaa47c7d53c10ff592751568b58"}
Oct 30 19:09:18.740312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:18.740255    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" podStartSLOduration=1.740235632 podStartE2EDuration="1.740235632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:18.73503829 +0000 UTC m=+1849.622955398" watchObservedRunningTime="2026-10-30 19:09:18.740235632 +0000 UTC m=+1849.628152721"
Oct 30 19:09:19.052449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.051882    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:19.052449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.052407    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:09:19.052734 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.052484    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 19:09:19.268013 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.267971    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5" podUID="52394b87-7cd4-4084-bc08-8e4e097173c8" containerName="payload-processing" containerID="cri-o://86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e" gracePeriod=25
Oct 30 19:09:19.391240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.391184    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n" podUID="ddf30c94-65c8-433c-9de7-7c3d5deb1719" containerName="payload-pre-processing" containerID="cri-o://2724c0237e350e9cf0a6d280d2390b4b3fbd103f0da49eb68ee0dd6f773edc93" gracePeriod=25
Oct 30 19:09:19.669262 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.668773    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:09:19.673048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.672644    4267 generic.go:356] "Generic (PLEG): container finished" podID="ddf30c94-65c8-433c-9de7-7c3d5deb1719" containerID="2724c0237e350e9cf0a6d280d2390b4b3fbd103f0da49eb68ee0dd6f773edc93" exitCode=0
Oct 30 19:09:19.673048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.672712    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n" event={"ID":"ddf30c94-65c8-433c-9de7-7c3d5deb1719","Type":"ContainerDied","Data":"2724c0237e350e9cf0a6d280d2390b4b3fbd103f0da49eb68ee0dd6f773edc93"}
Oct 30 19:09:19.675283 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.674390    4267 generic.go:356] "Generic (PLEG): container finished" podID="52394b87-7cd4-4084-bc08-8e4e097173c8" containerID="86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e" exitCode=0
Oct 30 19:09:19.675976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.675436    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"
Oct 30 19:09:19.675976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.675642    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5" event={"ID":"52394b87-7cd4-4084-bc08-8e4e097173c8","Type":"ContainerDied","Data":"86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e"}
Oct 30 19:09:19.675976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.675682    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5" event={"ID":"52394b87-7cd4-4084-bc08-8e4e097173c8","Type":"ContainerDied","Data":"4068a6ee42b3acc44ecf1b277417329aae454eb67f504b706372c70540eec186"}
Oct 30 19:09:19.675976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.675714    4267 scope.go:122] "RemoveContainer" containerID="86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e"
Oct 30 19:09:19.695814 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.695782    4267 scope.go:122] "RemoveContainer" containerID="86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e"
Oct 30 19:09:19.696252 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:09:19.696223    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e\": container with ID starting with 86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e not found: ID does not exist" containerID="86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e"
Oct 30 19:09:19.696328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.696264    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e"} err="failed to get container status \"86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e\": rpc error: code = NotFound desc = could not find container \"86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e\": container with ID starting with 86100eade8d0d2c84ff6fdc95aef4ca199022020f57806eb5e983a41708ff48e not found: ID does not exist"
Oct 30 19:09:19.789937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.789906    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:09:19.859516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.859162    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf30c94-65c8-433c-9de7-7c3d5deb1719-kube-api-access-gtk7m\" (UniqueName: \"kubernetes.io/projected/ddf30c94-65c8-433c-9de7-7c3d5deb1719-kube-api-access-gtk7m\") pod \"ddf30c94-65c8-433c-9de7-7c3d5deb1719\" (UID: \"ddf30c94-65c8-433c-9de7-7c3d5deb1719\") "
Oct 30 19:09:19.859516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.859374    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52394b87-7cd4-4084-bc08-8e4e097173c8-kube-api-access-mm84q\" (UniqueName: \"kubernetes.io/projected/52394b87-7cd4-4084-bc08-8e4e097173c8-kube-api-access-mm84q\") pod \"52394b87-7cd4-4084-bc08-8e4e097173c8\" (UID: \"52394b87-7cd4-4084-bc08-8e4e097173c8\") "
Oct 30 19:09:19.859516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.859423    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52394b87-7cd4-4084-bc08-8e4e097173c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52394b87-7cd4-4084-bc08-8e4e097173c8-plugins-config-volume\") pod \"52394b87-7cd4-4084-bc08-8e4e097173c8\" (UID: \"52394b87-7cd4-4084-bc08-8e4e097173c8\") "
Oct 30 19:09:19.859516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.859475    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf30c94-65c8-433c-9de7-7c3d5deb1719-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf30c94-65c8-433c-9de7-7c3d5deb1719-plugins-config-volume\") pod \"ddf30c94-65c8-433c-9de7-7c3d5deb1719\" (UID: \"ddf30c94-65c8-433c-9de7-7c3d5deb1719\") "
Oct 30 19:09:19.864240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.863777    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf30c94-65c8-433c-9de7-7c3d5deb1719-plugins-config-volume" pod "ddf30c94-65c8-433c-9de7-7c3d5deb1719" (UID: "ddf30c94-65c8-433c-9de7-7c3d5deb1719"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.864240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.863968    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52394b87-7cd4-4084-bc08-8e4e097173c8-plugins-config-volume" pod "52394b87-7cd4-4084-bc08-8e4e097173c8" (UID: "52394b87-7cd4-4084-bc08-8e4e097173c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.865707 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.865678    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52394b87-7cd4-4084-bc08-8e4e097173c8-kube-api-access-mm84q" pod "52394b87-7cd4-4084-bc08-8e4e097173c8" (UID: "52394b87-7cd4-4084-bc08-8e4e097173c8"). InnerVolumeSpecName "kube-api-access-mm84q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.867632 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.867601    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf30c94-65c8-433c-9de7-7c3d5deb1719-kube-api-access-gtk7m" pod "ddf30c94-65c8-433c-9de7-7c3d5deb1719" (UID: "ddf30c94-65c8-433c-9de7-7c3d5deb1719"). InnerVolumeSpecName "kube-api-access-gtk7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.962061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.961857    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtk7m\" (UniqueName: \"kubernetes.io/projected/ddf30c94-65c8-433c-9de7-7c3d5deb1719-kube-api-access-gtk7m\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.962061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.961890    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm84q\" (UniqueName: \"kubernetes.io/projected/52394b87-7cd4-4084-bc08-8e4e097173c8-kube-api-access-mm84q\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.962061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.961900    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52394b87-7cd4-4084-bc08-8e4e097173c8-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.962061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:19.961911    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf30c94-65c8-433c-9de7-7c3d5deb1719-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:09:20.083514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:20.066469    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:09:20.083514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:20.066552    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 19:09:20.151139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:20.150734    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"]
Oct 30 19:09:20.155541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:20.155512    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4501c8d4-644cfcbcd5-rjml5"]
Oct 30 19:09:20.180100 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:09:20.180065    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:09:20.685793 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:20.685764    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"
Oct 30 19:09:20.686013 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:20.685807    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n" event={"ID":"ddf30c94-65c8-433c-9de7-7c3d5deb1719","Type":"ContainerDied","Data":"97dad9bd84498f2c1da6218572fa75f371c03eb9f7ed188df36c3a89b2561fb6"}
Oct 30 19:09:20.686013 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:20.685886    4267 scope.go:122] "RemoveContainer" containerID="2724c0237e350e9cf0a6d280d2390b4b3fbd103f0da49eb68ee0dd6f773edc93"
Oct 30 19:09:20.857447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:20.857408    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"]
Oct 30 19:09:20.867055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:20.867016    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4501c8d4-5c5486779f-5vq7n"]
Oct 30 19:09:21.046076 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:21.045968    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:09:21.046076 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:21.046036    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 19:09:21.245234 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:21.245202    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52394b87-7cd4-4084-bc08-8e4e097173c8" path="/var/lib/kubelet/pods/52394b87-7cd4-4084-bc08-8e4e097173c8/volumes"
Oct 30 19:09:21.245748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:21.245730    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf30c94-65c8-433c-9de7-7c3d5deb1719" path="/var/lib/kubelet/pods/ddf30c94-65c8-433c-9de7-7c3d5deb1719/volumes"
Oct 30 19:09:22.052551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:22.045678    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:09:22.052551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:22.045743    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 19:09:23.046086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:23.046046    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:09:23.046287 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:23.046108    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 19:09:24.046338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:24.046297    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 19:09:24.046795 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:24.046373    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 19:09:25.052149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:25.052115    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:25.052761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:25.052583    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:25.053563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:25.053536    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:09:30.783878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:30.783846    4267 scope.go:122] "RemoveContainer" containerID="1fb72a0c118fc4e07bc50753b1fd86bf1cb130c1d79ffd39c021af9c56edcce6"
Oct 30 19:09:30.794336 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:30.794313    4267 scope.go:122] "RemoveContainer" containerID="c6006eb49f00b8f069b92f0a77307d639ff7ad1dc16d68738c8cc8d0e7ecbb53"
Oct 30 19:09:30.804541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:30.804520    4267 scope.go:122] "RemoveContainer" containerID="5d433f909dc6d8fbb94b51f1b2601c27257017c56090497f67240a36822ca5df"
Oct 30 19:09:30.815775 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:09:30.815748    4267 scope.go:122] "RemoveContainer" containerID="d38cd835ec59ddcdbdf62e1e73014f85d3458b6747059b1892e51341c4af5315"
Oct 30 19:09:50.161659 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:09:50.161608    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:10:01.683871 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.683820    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"]
Oct 30 19:10:01.684576 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.684557    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ddf30c94-65c8-433c-9de7-7c3d5deb1719" containerName="payload-pre-processing"
Oct 30 19:10:01.684655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.684576    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf30c94-65c8-433c-9de7-7c3d5deb1719" containerName="payload-pre-processing"
Oct 30 19:10:01.684655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.684600    4267 cpu_manager.go:414] "containerMap: removing container" podUID="52394b87-7cd4-4084-bc08-8e4e097173c8" containerName="payload-processing"
Oct 30 19:10:01.684655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.684605    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52394b87-7cd4-4084-bc08-8e4e097173c8" containerName="payload-processing"
Oct 30 19:10:01.684809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.684729    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="52394b87-7cd4-4084-bc08-8e4e097173c8" containerName="payload-processing"
Oct 30 19:10:01.684809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.684746    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf30c94-65c8-433c-9de7-7c3d5deb1719" containerName="payload-pre-processing"
Oct 30 19:10:01.688591 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.688564    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.729045 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.728995    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.729257 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.729099    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.729322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.729295    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.729384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.729338    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.729440 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.729428    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.729535 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.729505    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04ad0d0-e089-4dce-8735-968a6eae12e5-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.729722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.729698    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.729802 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.729748    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x54lr\" (UniqueName: \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-kube-api-access-x54lr\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.729877 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.729850    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.750761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.750718    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"]
Oct 30 19:10:01.809519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.794219    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ae6a138d-openshift-default-dockercfg-hrsvc\""
Oct 30 19:10:01.842204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.842164    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.842433 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.842239    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.842433 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.842354    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04ad0d0-e089-4dce-8735-968a6eae12e5-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.842633 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.842482    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.842709 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.842534    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x54lr\" (UniqueName: \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-kube-api-access-x54lr\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.843018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.842985    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.843150 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.843131    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.843203 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.843166    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.843368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.843351    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.877404 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.877349    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.882695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.880232    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04ad0d0-e089-4dce-8735-968a6eae12e5-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.882695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.880531    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.895154 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.895104    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.895629 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.895600    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.917594 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.914201    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.939291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.938645    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.939462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.939357    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x54lr\" (UniqueName: \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-kube-api-access-x54lr\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:01.940051 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:01.940026    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:02.004181 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:02.004144    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:02.455587 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:10:02.455552    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04ad0d0_e089_4dce_8735_968a6eae12e5.slice/crio-f30d9b04ea5913f3973278f9dec2a2496447ec18570bd893e6e7b2d3f333e2ec WatchSource:0}: Error finding container f30d9b04ea5913f3973278f9dec2a2496447ec18570bd893e6e7b2d3f333e2ec: Status 404 returned error can't find the container with id f30d9b04ea5913f3973278f9dec2a2496447ec18570bd893e6e7b2d3f333e2ec
Oct 30 19:10:02.456340 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:02.456315    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"]
Oct 30 19:10:02.458306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:02.458268    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:10:02.458418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:02.458355    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:10:02.458418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:02.458404    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:10:02.986106 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:02.986073    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"]
Oct 30 19:10:02.990626 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:02.990604    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.106885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.106559    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"]
Oct 30 19:10:03.152224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.150593    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.152224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.150698    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.152224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.150750    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.152224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.150835    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.152224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.150871    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.152224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.150895    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea86300-bcfd-4c31-893d-342130676629-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.152224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.150994    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea86300-bcfd-4c31-893d-342130676629-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.152224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.151083    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.152224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.151192    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzcrw\" (UniqueName: \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-kube-api-access-gzcrw\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.253960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.253005    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea86300-bcfd-4c31-893d-342130676629-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.253960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.253073    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.253960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.253160    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzcrw\" (UniqueName: \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-kube-api-access-gzcrw\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.253960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.253246    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.253960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.253316    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.253960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.253368    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.253960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.253461    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.253960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.253519    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.253960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.253544    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea86300-bcfd-4c31-893d-342130676629-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.254571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.254110    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-data\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.254571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.254143    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-credential-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.254571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.254393    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea86300-bcfd-4c31-893d-342130676629-istiod-ca-cert\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.254571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.254420    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-certs\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.254788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.254657    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-socket\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.260552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.260036    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea86300-bcfd-4c31-893d-342130676629-istio-podinfo\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.262564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.260938    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-envoy\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.282455 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.282420    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" event={"ID":"f04ad0d0-e089-4dce-8735-968a6eae12e5","Type":"ContainerStarted","Data":"86626663480cd2f650d2043e2f8f86dc167e523ba5eb866233cf3e74a9383e4e"}
Oct 30 19:10:03.282608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.282485    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" event={"ID":"f04ad0d0-e089-4dce-8735-968a6eae12e5","Type":"ContainerStarted","Data":"f30d9b04ea5913f3973278f9dec2a2496447ec18570bd893e6e7b2d3f333e2ec"}
Oct 30 19:10:03.342541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.341699    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-istio-token\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.345165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.345131    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzcrw\" (UniqueName: \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-kube-api-access-gzcrw\") pod \"e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") " pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:03.425383 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.425309    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" podStartSLOduration=2.425284292 podStartE2EDuration="2.425284292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:03.410712953 +0000 UTC m=+1894.298630045" watchObservedRunningTime="2026-10-30 19:10:03.425284292 +0000 UTC m=+1894.313201383"
Oct 30 19:10:03.606965 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:03.606872    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:04.008008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:04.007549    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:04.008008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:04.007687    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:10:04.008008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:04.007733    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 19:10:04.132517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:04.131899    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"]
Oct 30 19:10:04.164982 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:04.164238    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:10:04.164982 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:04.164371    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:10:04.164982 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:04.164413    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:10:04.298522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:04.297378    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" event={"ID":"fea86300-bcfd-4c31-893d-342130676629","Type":"ContainerStarted","Data":"6fac196f0e9f791d4719caee73b18dc1dd347f8cf1b0d092e82459a04ed4f54b"}
Oct 30 19:10:05.005183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:05.005136    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:10:05.005388 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:05.005194    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 19:10:05.308146 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:05.308062    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" event={"ID":"fea86300-bcfd-4c31-893d-342130676629","Type":"ContainerStarted","Data":"88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc"}
Oct 30 19:10:05.571402 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:05.571238    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" podStartSLOduration=3.5712199829999998 podStartE2EDuration="3.571219983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:05.50547148 +0000 UTC m=+1896.393388566" watchObservedRunningTime="2026-10-30 19:10:05.571219983 +0000 UTC m=+1896.459137069"
Oct 30 19:10:05.607548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:05.607516    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:05.612670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:05.612631    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:10:05.612965 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:05.612934    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" podUID="fea86300-bcfd-4c31-893d-342130676629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 19:10:06.005078 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:06.005040    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:10:06.005274 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:06.005099    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 19:10:06.607368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:06.607325    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:10:06.607851 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:06.607393    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" podUID="fea86300-bcfd-4c31-893d-342130676629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 19:10:07.005325 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:07.005280    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:10:07.005541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:07.005360    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 19:10:07.607894 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:07.607848    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:10:07.608317 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:07.607930    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" podUID="fea86300-bcfd-4c31-893d-342130676629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 19:10:08.005393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:08.005360    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:10:08.005642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:08.005619    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 19:10:08.607661 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:08.607320    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:10:08.607661 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:08.607389    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" podUID="fea86300-bcfd-4c31-893d-342130676629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 19:10:09.005504 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.005454    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:10:09.005947 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.005556    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 19:10:09.353159 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.353059    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"]
Oct 30 19:10:09.353345 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.353319    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy" containerID="cri-o://694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463" gracePeriod=30
Oct 30 19:10:09.401170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.401131    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"]
Oct 30 19:10:09.401400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.401374    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy" containerID="cri-o://b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e" gracePeriod=30
Oct 30 19:10:09.405937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.405910    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"]
Oct 30 19:10:09.406144 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.406121    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" containerID="cri-o://021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd" gracePeriod=30
Oct 30 19:10:09.414170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.414139    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"]
Oct 30 19:10:09.414614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.414565    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy" containerID="cri-o://35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946" gracePeriod=30
Oct 30 19:10:09.607291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.607201    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:10:09.607291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:09.607255    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" podUID="fea86300-bcfd-4c31-893d-342130676629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 19:10:10.023743 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:10.017781    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:10.023743 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:10.022370    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:10.033985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:10.033937    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:10:10.607337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:10.607299    4267 patch_prober.go:29] interesting pod/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:10:10.607610 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:10.607364    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" podUID="fea86300-bcfd-4c31-893d-342130676629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 19:10:11.104291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:11.104251    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:10:11.104809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:11.104325    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:10:11.626534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:11.626330    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:11.626887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:11.626863    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:11.627796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:11.627772    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:10:14.624711 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.624202    4267 patch_prober.go:29] interesting pod/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:10:14.624711 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.624268    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:10:14.891064 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.882809    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:10:14.891064 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.885328    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:10:14.974361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.974321    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-credential-socket\") pod \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") "
Oct 30 19:10:14.974559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.974411    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-data\") pod \"006a5157-fb29-4104-84a6-c9005120d2e6\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") "
Oct 30 19:10:14.974559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.974473    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-kube-api-access-gttvz\" (UniqueName: \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-kube-api-access-gttvz\") pod \"006a5157-fb29-4104-84a6-c9005120d2e6\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") "
Oct 30 19:10:14.975384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975354    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-data" pod "006a5157-fb29-4104-84a6-c9005120d2e6" (UID: "006a5157-fb29-4104-84a6-c9005120d2e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:14.975518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975448    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/006a5157-fb29-4104-84a6-c9005120d2e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006a5157-fb29-4104-84a6-c9005120d2e6-istio-podinfo\") pod \"006a5157-fb29-4104-84a6-c9005120d2e6\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") "
Oct 30 19:10:14.975883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975595    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-certs\") pod \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") "
Oct 30 19:10:14.975883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975629    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istiod-ca-cert\") pod \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") "
Oct 30 19:10:14.975883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975719    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-kube-api-access-2pctw\" (UniqueName: \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-kube-api-access-2pctw\") pod \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") "
Oct 30 19:10:14.975883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975767    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-podinfo\") pod \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") "
Oct 30 19:10:14.975883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975805    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006a5157-fb29-4104-84a6-c9005120d2e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006a5157-fb29-4104-84a6-c9005120d2e6-istiod-ca-cert\") pod \"006a5157-fb29-4104-84a6-c9005120d2e6\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") "
Oct 30 19:10:14.975883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975857    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-socket\") pod \"006a5157-fb29-4104-84a6-c9005120d2e6\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") "
Oct 30 19:10:14.975883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975885    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-istio-token\" (UniqueName: \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-istio-token\") pod \"006a5157-fb29-4104-84a6-c9005120d2e6\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") "
Oct 30 19:10:14.976284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975948    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-envoy\") pod \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") "
Oct 30 19:10:14.976284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.976051    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-socket\") pod \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") "
Oct 30 19:10:14.976284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.976100    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-envoy\") pod \"006a5157-fb29-4104-84a6-c9005120d2e6\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") "
Oct 30 19:10:14.976284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.976153    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-data\") pod \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") "
Oct 30 19:10:14.976284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.976193    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-certs\") pod \"006a5157-fb29-4104-84a6-c9005120d2e6\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") "
Oct 30 19:10:14.976284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.976253    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-token\" (UniqueName: \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-token\") pod \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\" (UID: \"c1bc8497-e8fc-491b-bf9b-7cf563607f3c\") "
Oct 30 19:10:14.976615 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.976289    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-credential-socket\") pod \"006a5157-fb29-4104-84a6-c9005120d2e6\" (UID: \"006a5157-fb29-4104-84a6-c9005120d2e6\") "
Oct 30 19:10:14.976737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.975370    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-credential-socket" pod "c1bc8497-e8fc-491b-bf9b-7cf563607f3c" (UID: "c1bc8497-e8fc-491b-bf9b-7cf563607f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:14.977043 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.977020    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-credential-socket" pod "006a5157-fb29-4104-84a6-c9005120d2e6" (UID: "006a5157-fb29-4104-84a6-c9005120d2e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:14.977664 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.977642    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:14.977794 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.977778    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:14.978365 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.978347    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:14.978520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.978449    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006a5157-fb29-4104-84a6-c9005120d2e6-istiod-ca-cert" pod "006a5157-fb29-4104-84a6-c9005120d2e6" (UID: "006a5157-fb29-4104-84a6-c9005120d2e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:14.978896 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.978862    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/006a5157-fb29-4104-84a6-c9005120d2e6-istio-podinfo" pod "006a5157-fb29-4104-84a6-c9005120d2e6" (UID: "006a5157-fb29-4104-84a6-c9005120d2e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:10:14.979527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.979227    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istiod-ca-cert" pod "c1bc8497-e8fc-491b-bf9b-7cf563607f3c" (UID: "c1bc8497-e8fc-491b-bf9b-7cf563607f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:14.979527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.979379    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-data" pod "c1bc8497-e8fc-491b-bf9b-7cf563607f3c" (UID: "c1bc8497-e8fc-491b-bf9b-7cf563607f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:14.979677 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.979587    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-socket" pod "c1bc8497-e8fc-491b-bf9b-7cf563607f3c" (UID: "c1bc8497-e8fc-491b-bf9b-7cf563607f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:14.986718 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.979793    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-certs" pod "006a5157-fb29-4104-84a6-c9005120d2e6" (UID: "006a5157-fb29-4104-84a6-c9005120d2e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:14.986718 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.982592    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-socket" pod "006a5157-fb29-4104-84a6-c9005120d2e6" (UID: "006a5157-fb29-4104-84a6-c9005120d2e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:14.986718 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.983161    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-certs" pod "c1bc8497-e8fc-491b-bf9b-7cf563607f3c" (UID: "c1bc8497-e8fc-491b-bf9b-7cf563607f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:14.986718 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.985191    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-token" pod "c1bc8497-e8fc-491b-bf9b-7cf563607f3c" (UID: "c1bc8497-e8fc-491b-bf9b-7cf563607f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:14.989843 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.989812    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-envoy" pod "006a5157-fb29-4104-84a6-c9005120d2e6" (UID: "006a5157-fb29-4104-84a6-c9005120d2e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:14.990631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.990594    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-envoy" pod "c1bc8497-e8fc-491b-bf9b-7cf563607f3c" (UID: "c1bc8497-e8fc-491b-bf9b-7cf563607f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:14.990738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.990664    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-kube-api-access-gttvz" pod "006a5157-fb29-4104-84a6-c9005120d2e6" (UID: "006a5157-fb29-4104-84a6-c9005120d2e6"). InnerVolumeSpecName "kube-api-access-gttvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:14.991047 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.990965    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:10:14.991138 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.991078    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-istio-token" pod "006a5157-fb29-4104-84a6-c9005120d2e6" (UID: "006a5157-fb29-4104-84a6-c9005120d2e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:14.992063 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.992036    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-kube-api-access-2pctw" pod "c1bc8497-e8fc-491b-bf9b-7cf563607f3c" (UID: "c1bc8497-e8fc-491b-bf9b-7cf563607f3c"). InnerVolumeSpecName "kube-api-access-2pctw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:14.993024 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.993003    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 19:10:14.993122 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:14.993099    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-podinfo" pod "c1bc8497-e8fc-491b-bf9b-7cf563607f3c" (UID: "c1bc8497-e8fc-491b-bf9b-7cf563607f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:10:15.079337 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079294    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-envoy\") pod \"0ebcad52-7747-4d1c-be69-61193c14e5e3\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") "
Oct 30 19:10:15.079552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079365    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-data\") pod \"8d7121f3-9966-49c8-9838-083262e11048\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") "
Oct 30 19:10:15.079552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079452    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-data\") pod \"0ebcad52-7747-4d1c-be69-61193c14e5e3\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") "
Oct 30 19:10:15.079697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079553    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-kube-api-access-wt2lh\" (UniqueName: \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-kube-api-access-wt2lh\") pod \"0ebcad52-7747-4d1c-be69-61193c14e5e3\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") "
Oct 30 19:10:15.079697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079609    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-credential-socket\") pod \"8d7121f3-9966-49c8-9838-083262e11048\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") "
Oct 30 19:10:15.079697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079654    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7121f3-9966-49c8-9838-083262e11048-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7121f3-9966-49c8-9838-083262e11048-istiod-ca-cert\") pod \"8d7121f3-9966-49c8-9838-083262e11048\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") "
Oct 30 19:10:15.079831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079694    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-data" pod "8d7121f3-9966-49c8-9838-083262e11048" (UID: "8d7121f3-9966-49c8-9838-083262e11048"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:15.079831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079760    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-credential-socket\") pod \"0ebcad52-7747-4d1c-be69-61193c14e5e3\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") "
Oct 30 19:10:15.079831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079765    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-data" pod "0ebcad52-7747-4d1c-be69-61193c14e5e3" (UID: "0ebcad52-7747-4d1c-be69-61193c14e5e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:15.079831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079809    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebcad52-7747-4d1c-be69-61193c14e5e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebcad52-7747-4d1c-be69-61193c14e5e3-istiod-ca-cert\") pod \"0ebcad52-7747-4d1c-be69-61193c14e5e3\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") "
Oct 30 19:10:15.079976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079851    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-envoy\") pod \"8d7121f3-9966-49c8-9838-083262e11048\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") "
Oct 30 19:10:15.079976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079892    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-certs\") pod \"0ebcad52-7747-4d1c-be69-61193c14e5e3\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") "
Oct 30 19:10:15.079976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079912    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-credential-socket" pod "8d7121f3-9966-49c8-9838-083262e11048" (UID: "8d7121f3-9966-49c8-9838-083262e11048"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:15.079976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.079942    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-certs\") pod \"8d7121f3-9966-49c8-9838-083262e11048\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") "
Oct 30 19:10:15.080111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080007    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d7121f3-9966-49c8-9838-083262e11048-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7121f3-9966-49c8-9838-083262e11048-istio-podinfo\") pod \"8d7121f3-9966-49c8-9838-083262e11048\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") "
Oct 30 19:10:15.080111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080034    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-podinfo\") pod \"0ebcad52-7747-4d1c-be69-61193c14e5e3\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") "
Oct 30 19:10:15.080111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080020    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-credential-socket" pod "0ebcad52-7747-4d1c-be69-61193c14e5e3" (UID: "0ebcad52-7747-4d1c-be69-61193c14e5e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:15.080111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080069    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-kube-api-access-428rs\" (UniqueName: \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-kube-api-access-428rs\") pod \"8d7121f3-9966-49c8-9838-083262e11048\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") "
Oct 30 19:10:15.080111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080103    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-token\" (UniqueName: \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-token\") pod \"0ebcad52-7747-4d1c-be69-61193c14e5e3\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") "
Oct 30 19:10:15.080277 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080106    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7121f3-9966-49c8-9838-083262e11048-istiod-ca-cert" pod "8d7121f3-9966-49c8-9838-083262e11048" (UID: "8d7121f3-9966-49c8-9838-083262e11048"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:15.080277 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080180    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-istio-token\" (UniqueName: \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-istio-token\") pod \"8d7121f3-9966-49c8-9838-083262e11048\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") "
Oct 30 19:10:15.080277 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080245    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-certs" pod "8d7121f3-9966-49c8-9838-083262e11048" (UID: "8d7121f3-9966-49c8-9838-083262e11048"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:15.080277 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080274    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-socket\") pod \"8d7121f3-9966-49c8-9838-083262e11048\" (UID: \"8d7121f3-9966-49c8-9838-083262e11048\") "
Oct 30 19:10:15.080416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080310    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-socket\") pod \"0ebcad52-7747-4d1c-be69-61193c14e5e3\" (UID: \"0ebcad52-7747-4d1c-be69-61193c14e5e3\") "
Oct 30 19:10:15.080416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080379    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-certs" pod "0ebcad52-7747-4d1c-be69-61193c14e5e3" (UID: "0ebcad52-7747-4d1c-be69-61193c14e5e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:15.080540 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080519    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebcad52-7747-4d1c-be69-61193c14e5e3-istiod-ca-cert" pod "0ebcad52-7747-4d1c-be69-61193c14e5e3" (UID: "0ebcad52-7747-4d1c-be69-61193c14e5e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:15.080815 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080786    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-socket" pod "8d7121f3-9966-49c8-9838-083262e11048" (UID: "8d7121f3-9966-49c8-9838-083262e11048"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:15.080994 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.080974    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-socket" pod "0ebcad52-7747-4d1c-be69-61193c14e5e3" (UID: "0ebcad52-7747-4d1c-be69-61193c14e5e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:15.081181 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081152    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081180    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081196    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081213    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081225    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7121f3-9966-49c8-9838-083262e11048-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081238    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pctw\" (UniqueName: \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-kube-api-access-2pctw\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081252    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081265    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006a5157-fb29-4104-84a6-c9005120d2e6-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081279    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081294    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081307    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081321    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebcad52-7747-4d1c-be69-61193c14e5e3-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081335    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081349    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081363    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081376    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081392    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081405    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081419    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006a5157-fb29-4104-84a6-c9005120d2e6-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081433    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1bc8497-e8fc-491b-bf9b-7cf563607f3c-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081446    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081460    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081474    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gttvz\" (UniqueName: \"kubernetes.io/projected/006a5157-fb29-4104-84a6-c9005120d2e6-kube-api-access-gttvz\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081504    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006a5157-fb29-4104-84a6-c9005120d2e6-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.081670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.081519    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.082996 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.082968    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-kube-api-access-wt2lh" pod "0ebcad52-7747-4d1c-be69-61193c14e5e3" (UID: "0ebcad52-7747-4d1c-be69-61193c14e5e3"). InnerVolumeSpecName "kube-api-access-wt2lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:15.083906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.083849    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-envoy" pod "8d7121f3-9966-49c8-9838-083262e11048" (UID: "8d7121f3-9966-49c8-9838-083262e11048"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:15.083906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.083866    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d7121f3-9966-49c8-9838-083262e11048-istio-podinfo" pod "8d7121f3-9966-49c8-9838-083262e11048" (UID: "8d7121f3-9966-49c8-9838-083262e11048"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:10:15.083906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.083898    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-istio-token" pod "8d7121f3-9966-49c8-9838-083262e11048" (UID: "8d7121f3-9966-49c8-9838-083262e11048"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:15.084113 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.083967    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-podinfo" pod "0ebcad52-7747-4d1c-be69-61193c14e5e3" (UID: "0ebcad52-7747-4d1c-be69-61193c14e5e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:10:15.084415 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.084391    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-token" pod "0ebcad52-7747-4d1c-be69-61193c14e5e3" (UID: "0ebcad52-7747-4d1c-be69-61193c14e5e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:15.084470 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.084455    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-envoy" pod "0ebcad52-7747-4d1c-be69-61193c14e5e3" (UID: "0ebcad52-7747-4d1c-be69-61193c14e5e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:15.085155 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.085136    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-kube-api-access-428rs" pod "8d7121f3-9966-49c8-9838-083262e11048" (UID: "8d7121f3-9966-49c8-9838-083262e11048"). InnerVolumeSpecName "kube-api-access-428rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:15.183601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.183568    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt2lh\" (UniqueName: \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-kube-api-access-wt2lh\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.183601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.183599    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7121f3-9966-49c8-9838-083262e11048-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.183850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.183614    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7121f3-9966-49c8-9838-083262e11048-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.183850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.183627    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.183850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.183637    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-428rs\" (UniqueName: \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-kube-api-access-428rs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.183850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.183646    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.183850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.183660    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7121f3-9966-49c8-9838-083262e11048-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.183850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.183671    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebcad52-7747-4d1c-be69-61193c14e5e3-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:15.457787 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.457620    4267 generic.go:356] "Generic (PLEG): container finished" podID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerID="021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd" exitCode=0
Oct 30 19:10:15.457787 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.457702    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" event={"ID":"0ebcad52-7747-4d1c-be69-61193c14e5e3","Type":"ContainerDied","Data":"021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd"}
Oct 30 19:10:15.457787 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.457763    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm" event={"ID":"0ebcad52-7747-4d1c-be69-61193c14e5e3","Type":"ContainerDied","Data":"0752db37a653b36d38f2326381cb060b1b54c916c9a4aef0d83f8c41e190af03"}
Oct 30 19:10:15.458062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.457794    4267 scope.go:122] "RemoveContainer" containerID="021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd"
Oct 30 19:10:15.458062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.457679    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"
Oct 30 19:10:15.464510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.460048    4267 generic.go:356] "Generic (PLEG): container finished" podID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerID="694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463" exitCode=0
Oct 30 19:10:15.464510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.460084    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" event={"ID":"c1bc8497-e8fc-491b-bf9b-7cf563607f3c","Type":"ContainerDied","Data":"694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463"}
Oct 30 19:10:15.464510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.460101    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"
Oct 30 19:10:15.464510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.460160    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4" event={"ID":"c1bc8497-e8fc-491b-bf9b-7cf563607f3c","Type":"ContainerDied","Data":"31f49bf01683f29f02c3ce07997cec49c28d38c0d0d68a07d004c077319a7b4f"}
Oct 30 19:10:15.464510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.461960    4267 generic.go:356] "Generic (PLEG): container finished" podID="006a5157-fb29-4104-84a6-c9005120d2e6" containerID="b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e" exitCode=0
Oct 30 19:10:15.464510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.462097    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" event={"ID":"006a5157-fb29-4104-84a6-c9005120d2e6","Type":"ContainerDied","Data":"b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e"}
Oct 30 19:10:15.464510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.462135    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf" event={"ID":"006a5157-fb29-4104-84a6-c9005120d2e6","Type":"ContainerDied","Data":"35465c69edc103092c7c097b65db24d2f820918fdc3869eefcaf47dc1f8edd86"}
Oct 30 19:10:15.464510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.462197    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"
Oct 30 19:10:15.470514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.470070    4267 generic.go:356] "Generic (PLEG): container finished" podID="8d7121f3-9966-49c8-9838-083262e11048" containerID="35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946" exitCode=0
Oct 30 19:10:15.470514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.470135    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" event={"ID":"8d7121f3-9966-49c8-9838-083262e11048","Type":"ContainerDied","Data":"35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946"}
Oct 30 19:10:15.470514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.470169    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz" event={"ID":"8d7121f3-9966-49c8-9838-083262e11048","Type":"ContainerDied","Data":"cf7024dc26aa639485956b7c79cc6d6cff0d798e73214ecc6c39d74bae4bedce"}
Oct 30 19:10:15.470514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.470220    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"
Oct 30 19:10:15.478003 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.477986    4267 scope.go:122] "RemoveContainer" containerID="021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd"
Oct 30 19:10:15.478321 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:10:15.478297    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd\": container with ID starting with 021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd not found: ID does not exist" containerID="021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd"
Oct 30 19:10:15.478372 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.478334    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd"} err="failed to get container status \"021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd\": rpc error: code = NotFound desc = could not find container \"021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd\": container with ID starting with 021ea5ce45f6f117b0c88f6a5af6c992894527a6a49060a83f9bb502e2f653cd not found: ID does not exist"
Oct 30 19:10:15.478372 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.478357    4267 scope.go:122] "RemoveContainer" containerID="694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463"
Oct 30 19:10:15.491918 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.491892    4267 scope.go:122] "RemoveContainer" containerID="694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463"
Oct 30 19:10:15.492323 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:10:15.492297    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463\": container with ID starting with 694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463 not found: ID does not exist" containerID="694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463"
Oct 30 19:10:15.492433 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.492327    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463"} err="failed to get container status \"694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463\": rpc error: code = NotFound desc = could not find container \"694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463\": container with ID starting with 694a22887c578b10b08814ddc9dbd741497f20b45b5699fc75db28ebcf7c9463 not found: ID does not exist"
Oct 30 19:10:15.492433 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.492347    4267 scope.go:122] "RemoveContainer" containerID="b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e"
Oct 30 19:10:15.509640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.509615    4267 scope.go:122] "RemoveContainer" containerID="b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e"
Oct 30 19:10:15.509962 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:10:15.509942    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e\": container with ID starting with b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e not found: ID does not exist" containerID="b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e"
Oct 30 19:10:15.510059 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.509971    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e"} err="failed to get container status \"b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e\": rpc error: code = NotFound desc = could not find container \"b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e\": container with ID starting with b62755be9b0316f4119ede0ef2e882ce3e7d49f0d72a29f09ff52eea19cc867e not found: ID does not exist"
Oct 30 19:10:15.510059 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.509991    4267 scope.go:122] "RemoveContainer" containerID="35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946"
Oct 30 19:10:15.520147 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.520129    4267 scope.go:122] "RemoveContainer" containerID="35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946"
Oct 30 19:10:15.520453 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:10:15.520433    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946\": container with ID starting with 35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946 not found: ID does not exist" containerID="35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946"
Oct 30 19:10:15.520544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.520459    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946"} err="failed to get container status \"35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946\": rpc error: code = NotFound desc = could not find container \"35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946\": container with ID starting with 35a8f6db40da5cf0c1e178e63f7a814299f13d79a9f96f6c6f61d73e227cf946 not found: ID does not exist"
Oct 30 19:10:15.628182 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.628136    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"]
Oct 30 19:10:15.632630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.632599    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-dnhmm"]
Oct 30 19:10:15.738484 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.738395    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"]
Oct 30 19:10:15.743804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.743776    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-wtzc4"]
Oct 30 19:10:15.863755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.863710    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"]
Oct 30 19:10:15.871274 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.871236    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-pnqxz"]
Oct 30 19:10:15.977238 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.976755    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"]
Oct 30 19:10:15.981978 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:15.981939    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-975cf5-openshift-default-5fcdb49bfc-l5tqf"]
Oct 30 19:10:17.224973 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:17.224932    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" path="/var/lib/kubelet/pods/006a5157-fb29-4104-84a6-c9005120d2e6/volumes"
Oct 30 19:10:17.225558 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:17.225541    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" path="/var/lib/kubelet/pods/0ebcad52-7747-4d1c-be69-61193c14e5e3/volumes"
Oct 30 19:10:17.226004 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:17.225988    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7121f3-9966-49c8-9838-083262e11048" path="/var/lib/kubelet/pods/8d7121f3-9966-49c8-9838-083262e11048/volumes"
Oct 30 19:10:17.226781 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:17.226763    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" path="/var/lib/kubelet/pods/c1bc8497-e8fc-491b-bf9b-7cf563607f3c/volumes"
Oct 30 19:10:17.969074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:17.969039    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"]
Oct 30 19:10:18.018619 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:18.018580    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4501c8d4-9wxn6"]
Oct 30 19:10:19.224693 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:19.224658    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134" path="/var/lib/kubelet/pods/0f8bcdd4-9b3f-4a44-a2e9-af6c4e76d134/volumes"
Oct 30 19:10:20.162912 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:10:20.162870    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:10:30.884701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:30.884670    4267 scope.go:122] "RemoveContainer" containerID="fd32e8e074d56d586965fa96c9b5f74a2099827c6f40729df16eb86ad9eae986"
Oct 30 19:10:32.768144 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.768102    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"]
Oct 30 19:10:32.768803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.768787    4267 cpu_manager.go:414] "containerMap: removing container" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy"
Oct 30 19:10:32.768803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.768802    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy"
Oct 30 19:10:32.768890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.768818    4267 cpu_manager.go:414] "containerMap: removing container" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy"
Oct 30 19:10:32.768890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.768824    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy"
Oct 30 19:10:32.768890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.768862    4267 cpu_manager.go:414] "containerMap: removing container" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy"
Oct 30 19:10:32.768890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.768868    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy"
Oct 30 19:10:32.768890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.768888    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy"
Oct 30 19:10:32.769038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.768893    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy"
Oct 30 19:10:32.769038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.768997    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7121f3-9966-49c8-9838-083262e11048" containerName="istio-proxy"
Oct 30 19:10:32.769038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.769005    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebcad52-7747-4d1c-be69-61193c14e5e3" containerName="istio-proxy"
Oct 30 19:10:32.769038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.769017    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="006a5157-fb29-4104-84a6-c9005120d2e6" containerName="istio-proxy"
Oct 30 19:10:32.769038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.769025    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1bc8497-e8fc-491b-bf9b-7cf563607f3c" containerName="istio-proxy"
Oct 30 19:10:32.773999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.773980    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:32.908024 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.907984    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-24e88a-openshift-default-dockercfg-z98c9\""
Oct 30 19:10:32.976515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.975517    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-data\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:32.976515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.975609    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-podinfo\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:32.976515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.975693    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-certs\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:32.976515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.975766    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istiod-ca-cert\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:32.976515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.975799    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-credential-socket\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:32.976515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.975835    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-socket\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:32.976515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.975903    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-envoy\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:32.976515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.976000    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mlx\" (UniqueName: \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-kube-api-access-48mlx\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:32.976515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:32.976045    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-token\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.024570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.001521    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"]
Oct 30 19:10:33.077834 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.077798    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mlx\" (UniqueName: \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-kube-api-access-48mlx\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.078035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.077872    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-token\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.078035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.077922    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-data\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.078035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.077997    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-podinfo\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.078203 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.078097    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-certs\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.078203 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.078189    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istiod-ca-cert\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.078311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.078229    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-credential-socket\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.078311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.078267    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-socket\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.078412 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.078352    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-envoy\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.078466 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.078412    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-data\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.079140 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.079108    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-credential-socket\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.079420 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.079398    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-socket\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.079563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.079478    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istiod-ca-cert\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.079829 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.079806    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-certs\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.081637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.081614    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-envoy\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.082011 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.081986    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-podinfo\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.135183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.135151    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-token\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.135371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.135343    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mlx\" (UniqueName: \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-kube-api-access-48mlx\") pod \"e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") " pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.387767 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.387666    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:33.855699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.855645    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:10:33.856236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.856211    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:10:33.856399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.856379    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:10:33.864432 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:33.864399    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"]
Oct 30 19:10:34.631154 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.631117    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"]
Oct 30 19:10:34.636993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.636966    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:10:34.656142 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.656112    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b31d077-8101-44f4-86a6-c03845acda3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s\" (UID: \"8b31d077-8101-44f4-86a6-c03845acda3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:10:34.656367 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.656348    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhd6\" (UniqueName: \"kubernetes.io/projected/8b31d077-8101-44f4-86a6-c03845acda3e-kube-api-access-crhd6\") pod \"payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s\" (UID: \"8b31d077-8101-44f4-86a6-c03845acda3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:10:34.692065 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.692027    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"]
Oct 30 19:10:34.744025 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.739918    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-322e97-dockercfg-fhp54\""
Oct 30 19:10:34.744025 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.740005    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-322e97\""
Oct 30 19:10:34.761890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.759354    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b31d077-8101-44f4-86a6-c03845acda3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s\" (UID: \"8b31d077-8101-44f4-86a6-c03845acda3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:10:34.761890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.759444    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhd6\" (UniqueName: \"kubernetes.io/projected/8b31d077-8101-44f4-86a6-c03845acda3e-kube-api-access-crhd6\") pod \"payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s\" (UID: \"8b31d077-8101-44f4-86a6-c03845acda3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:10:34.761890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.760860    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b31d077-8101-44f4-86a6-c03845acda3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s\" (UID: \"8b31d077-8101-44f4-86a6-c03845acda3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:10:34.765928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.765878    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" event={"ID":"60ce3cd7-8197-4758-bfc1-cff2f15d12ae","Type":"ContainerStarted","Data":"aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75"}
Oct 30 19:10:34.767215 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.767172    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" event={"ID":"60ce3cd7-8197-4758-bfc1-cff2f15d12ae","Type":"ContainerStarted","Data":"ab6e2e80d7edd4d93f09284008dde5dc78335e761214692de4b0ac42feb700d4"}
Oct 30 19:10:34.797180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.797130    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"]
Oct 30 19:10:34.804950 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.804920    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:10:34.853977 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.851277    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"]
Oct 30 19:10:34.862315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.861366    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph5r5\" (UniqueName: \"kubernetes.io/projected/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-kube-api-access-ph5r5\") pod \"payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk\" (UID: \"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229\") " pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:10:34.862315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.861454    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk\" (UID: \"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229\") " pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:10:34.872513 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.869246    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhd6\" (UniqueName: \"kubernetes.io/projected/8b31d077-8101-44f4-86a6-c03845acda3e-kube-api-access-crhd6\") pod \"payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s\" (UID: \"8b31d077-8101-44f4-86a6-c03845acda3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:10:34.892351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.892197    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" podStartSLOduration=2.892174319 podStartE2EDuration="2.892174319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:34.888961264 +0000 UTC m=+1925.776878354" watchObservedRunningTime="2026-10-30 19:10:34.892174319 +0000 UTC m=+1925.780091411"
Oct 30 19:10:34.966571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.954933    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:10:34.970826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.970780    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph5r5\" (UniqueName: \"kubernetes.io/projected/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-kube-api-access-ph5r5\") pod \"payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk\" (UID: \"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229\") " pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:10:34.971134 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.971111    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk\" (UID: \"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229\") " pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:10:34.972298 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:34.972274    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk\" (UID: \"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229\") " pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:10:35.090252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.090215    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph5r5\" (UniqueName: \"kubernetes.io/projected/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-kube-api-access-ph5r5\") pod \"payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk\" (UID: \"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229\") " pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:10:35.138351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.137779    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:10:35.387900 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.387853    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:35.393811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.393782    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:35.456131 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:10:35.456095    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b31d077_8101_44f4_86a6_c03845acda3e.slice/crio-f1a0952e44c86bc054155a2170c65dbc8f3bc63f07267aaeb9eedbade3676696 WatchSource:0}: Error finding container f1a0952e44c86bc054155a2170c65dbc8f3bc63f07267aaeb9eedbade3676696: Status 404 returned error can't find the container with id f1a0952e44c86bc054155a2170c65dbc8f3bc63f07267aaeb9eedbade3676696
Oct 30 19:10:35.458870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.458843    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"]
Oct 30 19:10:35.597805 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:10:35.597776    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90234b3d_9ec5_4d5d_b1ee_8b517d3c5229.slice/crio-210572982a06d7b0445965f8c48ec03d25893c568a355cf08d33bb74f9a64340 WatchSource:0}: Error finding container 210572982a06d7b0445965f8c48ec03d25893c568a355cf08d33bb74f9a64340: Status 404 returned error can't find the container with id 210572982a06d7b0445965f8c48ec03d25893c568a355cf08d33bb74f9a64340
Oct 30 19:10:35.598894 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.598865    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"]
Oct 30 19:10:35.779558 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.779461    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s" event={"ID":"8b31d077-8101-44f4-86a6-c03845acda3e","Type":"ContainerStarted","Data":"aef33eab280b5904ed2ed25246d5513fa7bbf4af9dc4eb99d6d7750f359fc4d8"}
Oct 30 19:10:35.779710 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.779564    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:10:35.779710 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.779588    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s" event={"ID":"8b31d077-8101-44f4-86a6-c03845acda3e","Type":"ContainerStarted","Data":"f1a0952e44c86bc054155a2170c65dbc8f3bc63f07267aaeb9eedbade3676696"}
Oct 30 19:10:35.781186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.781157    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk" event={"ID":"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229","Type":"ContainerStarted","Data":"bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0"}
Oct 30 19:10:35.781316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.781209    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk" event={"ID":"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229","Type":"ContainerStarted","Data":"210572982a06d7b0445965f8c48ec03d25893c568a355cf08d33bb74f9a64340"}
Oct 30 19:10:35.781413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.781392    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:35.781455 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.781436    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:10:35.782455 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.782427    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:10:35.898295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:35.898238    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s" podStartSLOduration=1.898220533 podStartE2EDuration="1.898220533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:35.893372645 +0000 UTC m=+1926.781289750" watchObservedRunningTime="2026-10-30 19:10:35.898220533 +0000 UTC m=+1926.786137623"
Oct 30 19:10:36.159213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:36.159107    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk" podStartSLOduration=2.159091579 podStartE2EDuration="2.159091579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:36.153439785 +0000 UTC m=+1927.041356875" watchObservedRunningTime="2026-10-30 19:10:36.159091579 +0000 UTC m=+1927.047008669"
Oct 30 19:10:42.494560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.487813    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"]
Oct 30 19:10:42.494560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.493545    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:10:42.593213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.593176    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-24e88a\""
Oct 30 19:10:42.593675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.593652    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-24e88a-dockercfg-2dzbf\""
Oct 30 19:10:42.673596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.673373    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62500e8-c362-4892-a8b8-003da67a4112-plugins-config-volume\") pod \"payload-processing-e2e-api-b-24e88a-597b894b45-mflg7\" (UID: \"d62500e8-c362-4892-a8b8-003da67a4112\") " pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:10:42.676677 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.676602    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7zsh\" (UniqueName: \"kubernetes.io/projected/d62500e8-c362-4892-a8b8-003da67a4112-kube-api-access-w7zsh\") pod \"payload-processing-e2e-api-b-24e88a-597b894b45-mflg7\" (UID: \"d62500e8-c362-4892-a8b8-003da67a4112\") " pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:10:42.712045 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.712008    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"]
Oct 30 19:10:42.769157 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.769055    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"]
Oct 30 19:10:42.779114 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.779074    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62500e8-c362-4892-a8b8-003da67a4112-plugins-config-volume\") pod \"payload-processing-e2e-api-b-24e88a-597b894b45-mflg7\" (UID: \"d62500e8-c362-4892-a8b8-003da67a4112\") " pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:10:42.779300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.779281    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7zsh\" (UniqueName: \"kubernetes.io/projected/d62500e8-c362-4892-a8b8-003da67a4112-kube-api-access-w7zsh\") pod \"payload-processing-e2e-api-b-24e88a-597b894b45-mflg7\" (UID: \"d62500e8-c362-4892-a8b8-003da67a4112\") " pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:10:42.779621 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.779598    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"]
Oct 30 19:10:42.779725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.779709    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:10:42.780097 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.780067    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62500e8-c362-4892-a8b8-003da67a4112-plugins-config-volume\") pod \"payload-processing-e2e-api-b-24e88a-597b894b45-mflg7\" (UID: \"d62500e8-c362-4892-a8b8-003da67a4112\") " pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:10:42.837276 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.837181    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7zsh\" (UniqueName: \"kubernetes.io/projected/d62500e8-c362-4892-a8b8-003da67a4112-kube-api-access-w7zsh\") pod \"payload-processing-e2e-api-b-24e88a-597b894b45-mflg7\" (UID: \"d62500e8-c362-4892-a8b8-003da67a4112\") " pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:10:42.881325 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.881286    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5\" (UID: \"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:10:42.881612 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.881586    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cshd\" (UniqueName: \"kubernetes.io/projected/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-kube-api-access-5cshd\") pod \"payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5\" (UID: \"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:10:42.984348 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.984307    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5\" (UID: \"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:10:42.984557 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.984543    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cshd\" (UniqueName: \"kubernetes.io/projected/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-kube-api-access-5cshd\") pod \"payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5\" (UID: \"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:10:42.985065 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:42.985045    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5\" (UID: \"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:10:43.044331 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.044256    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cshd\" (UniqueName: \"kubernetes.io/projected/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-kube-api-access-5cshd\") pod \"payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5\" (UID: \"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:10:43.096907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.096867    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:10:43.124837 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.124659    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:10:43.487618 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.487584    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"]
Oct 30 19:10:43.524613 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:10:43.524573    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf014bc_a4db_4af3_bd2f_14f9ad23f6fe.slice/crio-086dbdb6dcb3b69881c7c0d9af67d61e9224a5e413dc60d903bcbb71c6afcbc8 WatchSource:0}: Error finding container 086dbdb6dcb3b69881c7c0d9af67d61e9224a5e413dc60d903bcbb71c6afcbc8: Status 404 returned error can't find the container with id 086dbdb6dcb3b69881c7c0d9af67d61e9224a5e413dc60d903bcbb71c6afcbc8
Oct 30 19:10:43.539042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.539014    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"]
Oct 30 19:10:43.541474 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:10:43.541430    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62500e8_c362_4892_a8b8_003da67a4112.slice/crio-0a5861110ec3849722d5158077533384ff81431c68b6ce9da94b3472ac5799c3 WatchSource:0}: Error finding container 0a5861110ec3849722d5158077533384ff81431c68b6ce9da94b3472ac5799c3: Status 404 returned error can't find the container with id 0a5861110ec3849722d5158077533384ff81431c68b6ce9da94b3472ac5799c3
Oct 30 19:10:43.852474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.852367    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7" event={"ID":"d62500e8-c362-4892-a8b8-003da67a4112","Type":"ContainerStarted","Data":"f27ffe7b1b52db165fcf3964600fe55499470366cad8dc39fae0ac3609632f86"}
Oct 30 19:10:43.852474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.852435    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7" event={"ID":"d62500e8-c362-4892-a8b8-003da67a4112","Type":"ContainerStarted","Data":"0a5861110ec3849722d5158077533384ff81431c68b6ce9da94b3472ac5799c3"}
Oct 30 19:10:43.852474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.852467    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:10:43.855029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.854997    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5" event={"ID":"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe","Type":"ContainerStarted","Data":"6cc7dfb7315181b229ae984ec8edf23eeedc5934bbedf2e55a11fcec9cd1a782"}
Oct 30 19:10:43.855182 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.855053    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:10:43.855182 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.855076    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5" event={"ID":"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe","Type":"ContainerStarted","Data":"086dbdb6dcb3b69881c7c0d9af67d61e9224a5e413dc60d903bcbb71c6afcbc8"}
Oct 30 19:10:43.971898 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:43.971837    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7" podStartSLOduration=1.971822354 podStartE2EDuration="1.971822354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:43.96760332 +0000 UTC m=+1934.855520408" watchObservedRunningTime="2026-10-30 19:10:43.971822354 +0000 UTC m=+1934.859739480"
Oct 30 19:10:44.107756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:44.102341    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5" podStartSLOduration=2.102317005 podStartE2EDuration="2.102317005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:44.08941694 +0000 UTC m=+1934.977334031" watchObservedRunningTime="2026-10-30 19:10:44.102317005 +0000 UTC m=+1934.990234094"
Oct 30 19:10:46.791890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:46.791858    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:10:46.792371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:46.791933    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:10:50.196839 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:10:50.196686    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:10:54.501052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:54.500884    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"]
Oct 30 19:10:54.506950 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:54.506890    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy" containerID="cri-o://62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3" gracePeriod=30
Oct 30 19:10:54.868716 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:54.868634    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:10:54.868716 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:54.868709    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:10:55.702575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:55.702531    4267 patch_prober.go:29] interesting pod/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:10:55.702987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:55.702601    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:10:59.805830 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.805802    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 19:10:59.860382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860338    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istiod-ca-cert\") pod \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") "
Oct 30 19:10:59.860555 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860476    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-credential-socket\") pod \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") "
Oct 30 19:10:59.860555 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860525    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-kube-api-access-vjl2w\" (UniqueName: \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-kube-api-access-vjl2w\") pod \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") "
Oct 30 19:10:59.860647 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860588    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-envoy\") pod \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") "
Oct 30 19:10:59.860684 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860660    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-token\" (UniqueName: \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-token\") pod \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") "
Oct 30 19:10:59.860731 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860692    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-certs\") pod \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") "
Oct 30 19:10:59.860790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860732    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-podinfo\") pod \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") "
Oct 30 19:10:59.860790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860771    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-credential-socket" pod "4a3a967e-d63e-46e5-ba22-292d2c997aa6" (UID: "4a3a967e-d63e-46e5-ba22-292d2c997aa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:59.860903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860805    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istiod-ca-cert" pod "4a3a967e-d63e-46e5-ba22-292d2c997aa6" (UID: "4a3a967e-d63e-46e5-ba22-292d2c997aa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:59.860903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860834    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-data\") pod \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") "
Oct 30 19:10:59.860903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.860879    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-socket\") pod \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\" (UID: \"4a3a967e-d63e-46e5-ba22-292d2c997aa6\") "
Oct 30 19:10:59.861065 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.861036    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-certs" pod "4a3a967e-d63e-46e5-ba22-292d2c997aa6" (UID: "4a3a967e-d63e-46e5-ba22-292d2c997aa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:59.861294 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.861265    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-socket" pod "4a3a967e-d63e-46e5-ba22-292d2c997aa6" (UID: "4a3a967e-d63e-46e5-ba22-292d2c997aa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:59.861385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.861319    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-data" pod "4a3a967e-d63e-46e5-ba22-292d2c997aa6" (UID: "4a3a967e-d63e-46e5-ba22-292d2c997aa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:59.861737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.861712    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.861737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.861745    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.861900 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.861761    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.861900 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.861775    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.861900 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.861791    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.863528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.862957    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-envoy" pod "4a3a967e-d63e-46e5-ba22-292d2c997aa6" (UID: "4a3a967e-d63e-46e5-ba22-292d2c997aa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:59.863528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.863179    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-podinfo" pod "4a3a967e-d63e-46e5-ba22-292d2c997aa6" (UID: "4a3a967e-d63e-46e5-ba22-292d2c997aa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:10:59.863528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.863308    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-token" pod "4a3a967e-d63e-46e5-ba22-292d2c997aa6" (UID: "4a3a967e-d63e-46e5-ba22-292d2c997aa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:59.863528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.863343    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-kube-api-access-vjl2w" pod "4a3a967e-d63e-46e5-ba22-292d2c997aa6" (UID: "4a3a967e-d63e-46e5-ba22-292d2c997aa6"). InnerVolumeSpecName "kube-api-access-vjl2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:59.963888 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.963836    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjl2w\" (UniqueName: \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-kube-api-access-vjl2w\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.963888 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.963895    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.964108 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.963911    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.964108 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:10:59.963927    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3a967e-d63e-46e5-ba22-292d2c997aa6-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:00.000740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:00.000704    4267 generic.go:356] "Generic (PLEG): container finished" podID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerID="62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3" exitCode=0
Oct 30 19:11:00.000930 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:00.000779    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"
Oct 30 19:11:00.000992 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:00.000969    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" event={"ID":"4a3a967e-d63e-46e5-ba22-292d2c997aa6","Type":"ContainerDied","Data":"62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3"}
Oct 30 19:11:00.001043 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:00.001021    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn" event={"ID":"4a3a967e-d63e-46e5-ba22-292d2c997aa6","Type":"ContainerDied","Data":"288c5427ebaab80fc1d1fe45bd10c44557dff114eed209f22cf64d45f927138d"}
Oct 30 19:11:00.001089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:00.001067    4267 scope.go:122] "RemoveContainer" containerID="62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3"
Oct 30 19:11:00.017423 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:00.017401    4267 scope.go:122] "RemoveContainer" containerID="62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3"
Oct 30 19:11:00.017766 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:11:00.017739    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3\": container with ID starting with 62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3 not found: ID does not exist" containerID="62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3"
Oct 30 19:11:00.017857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:00.017776    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3"} err="failed to get container status \"62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3\": rpc error: code = NotFound desc = could not find container \"62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3\": container with ID starting with 62d38412885db5caa082943f207903caa6b02165a080dab9dceaa434379afcb3 not found: ID does not exist"
Oct 30 19:11:00.180351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:00.180271    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"]
Oct 30 19:11:00.184290 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:00.184264    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-2n2tn"]
Oct 30 19:11:01.225057 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:01.225018    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" path="/var/lib/kubelet/pods/4a3a967e-d63e-46e5-ba22-292d2c997aa6/volumes"
Oct 30 19:11:20.159950 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:11:20.159912    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:11:22.360314 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.360273    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"]
Oct 30 19:11:22.361043 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.361026    4267 cpu_manager.go:414] "containerMap: removing container" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy"
Oct 30 19:11:22.361088 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.361044    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy"
Oct 30 19:11:22.361183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.361168    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3a967e-d63e-46e5-ba22-292d2c997aa6" containerName="istio-proxy"
Oct 30 19:11:22.364714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.364697    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:22.421394 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.421342    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"]
Oct 30 19:11:22.463591 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.463560    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:11:22.463784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.463615    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:11:22.473105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.473082    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:22.473266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.473249    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qdcb\" (UniqueName: \"kubernetes.io/projected/e746d4ba-25cf-4492-a900-71f8702ae6e9-kube-api-access-2qdcb\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:22.473340 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.473308    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:22.575183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.575151    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qdcb\" (UniqueName: \"kubernetes.io/projected/e746d4ba-25cf-4492-a900-71f8702ae6e9-kube-api-access-2qdcb\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:22.575354 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.575216    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:22.575402 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.575352    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:22.575923 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.575899    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:22.577769 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.577747    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:22.631938 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.631871    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qdcb\" (UniqueName: \"kubernetes.io/projected/e746d4ba-25cf-4492-a900-71f8702ae6e9-kube-api-access-2qdcb\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:22.678603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:22.678573    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:23.062255 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:23.062222    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"]
Oct 30 19:11:23.063119 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:11:23.063093    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode746d4ba_25cf_4492_a900_71f8702ae6e9.slice/crio-80b6b784f5f235c55b82fe5da7e7ac40ecfb50d59b00328e22b497fea40fa059 WatchSource:0}: Error finding container 80b6b784f5f235c55b82fe5da7e7ac40ecfb50d59b00328e22b497fea40fa059: Status 404 returned error can't find the container with id 80b6b784f5f235c55b82fe5da7e7ac40ecfb50d59b00328e22b497fea40fa059
Oct 30 19:11:23.274931 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:23.274898    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw" event={"ID":"e746d4ba-25cf-4492-a900-71f8702ae6e9","Type":"ContainerStarted","Data":"80b6b784f5f235c55b82fe5da7e7ac40ecfb50d59b00328e22b497fea40fa059"}
Oct 30 19:11:24.284963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:24.284879    4267 generic.go:356] "Generic (PLEG): container finished" podID="e746d4ba-25cf-4492-a900-71f8702ae6e9" containerID="201165b380506633435774868608a5029be8081206fcf3e9614f5150e0d64053" exitCode=0
Oct 30 19:11:24.285336 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:24.284963    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw" event={"ID":"e746d4ba-25cf-4492-a900-71f8702ae6e9","Type":"ContainerDied","Data":"201165b380506633435774868608a5029be8081206fcf3e9614f5150e0d64053"}
Oct 30 19:11:25.472718 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:25.472691    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:25.522658 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:25.522566    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-xks-ca\") pod \"e746d4ba-25cf-4492-a900-71f8702ae6e9\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") "
Oct 30 19:11:25.522658 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:25.522663    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e746d4ba-25cf-4492-a900-71f8702ae6e9-kube-api-access-2qdcb\" (UniqueName: \"kubernetes.io/projected/e746d4ba-25cf-4492-a900-71f8702ae6e9-kube-api-access-2qdcb\") pod \"e746d4ba-25cf-4492-a900-71f8702ae6e9\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") "
Oct 30 19:11:25.523004 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:25.522979    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-openshift-ca\") pod \"e746d4ba-25cf-4492-a900-71f8702ae6e9\" (UID: \"e746d4ba-25cf-4492-a900-71f8702ae6e9\") "
Oct 30 19:11:25.523780 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:25.523421    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-openshift-ca" pod "e746d4ba-25cf-4492-a900-71f8702ae6e9" (UID: "e746d4ba-25cf-4492-a900-71f8702ae6e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:25.523780 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:25.523749    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:25.525074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:25.525047    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e746d4ba-25cf-4492-a900-71f8702ae6e9-kube-api-access-2qdcb" pod "e746d4ba-25cf-4492-a900-71f8702ae6e9" (UID: "e746d4ba-25cf-4492-a900-71f8702ae6e9"). InnerVolumeSpecName "kube-api-access-2qdcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:25.525213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:25.525187    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-xks-ca" pod "e746d4ba-25cf-4492-a900-71f8702ae6e9" (UID: "e746d4ba-25cf-4492-a900-71f8702ae6e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:25.626258 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:25.626152    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e746d4ba-25cf-4492-a900-71f8702ae6e9-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:25.626258 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:25.626204    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qdcb\" (UniqueName: \"kubernetes.io/projected/e746d4ba-25cf-4492-a900-71f8702ae6e9-kube-api-access-2qdcb\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:26.301907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:26.301873    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"
Oct 30 19:11:26.301907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:26.301900    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw" event={"ID":"e746d4ba-25cf-4492-a900-71f8702ae6e9","Type":"ContainerDied","Data":"80b6b784f5f235c55b82fe5da7e7ac40ecfb50d59b00328e22b497fea40fa059"}
Oct 30 19:11:26.302127 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:26.301949    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80b6b784f5f235c55b82fe5da7e7ac40ecfb50d59b00328e22b497fea40fa059"
Oct 30 19:11:30.935779 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:30.935738    4267 scope.go:122] "RemoveContainer" containerID="dbf1f987f66e1b0ef1702d2ab9d1cbf116b7cf523f9fccb1272c3cc1992ccc25"
Oct 30 19:11:38.658677 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:38.658638    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"]
Oct 30 19:11:43.660300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:43.660248    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq" podUID="482f2d86-6da5-48b0-a8ff-52f3db5be0f1" containerName="payload-processing" containerID="cri-o://c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775" gracePeriod=25
Oct 30 19:11:44.887722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.887674    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"]
Oct 30 19:11:44.890972 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.890941    4267 cpu_manager.go:414] "containerMap: removing container" podUID="e746d4ba-25cf-4492-a900-71f8702ae6e9" containerName="revoke-keys"
Oct 30 19:11:44.891160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.891147    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e746d4ba-25cf-4492-a900-71f8702ae6e9" containerName="revoke-keys"
Oct 30 19:11:44.891469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.891455    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="e746d4ba-25cf-4492-a900-71f8702ae6e9" containerName="revoke-keys"
Oct 30 19:11:44.900815 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.900771    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:44.932222 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.932184    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"]
Oct 30 19:11:44.974940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.974907    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:11:44.975194 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.975084    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:11:44.978585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.978555    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42d5w\" (UniqueName: \"kubernetes.io/projected/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-kube-api-access-42d5w\") pod \"maas-api-revoke-sub-6ebe4e56-jf4gh\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:44.979060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.979032    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ebe4e56-jf4gh\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:44.979181 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:44.979094    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ebe4e56-jf4gh\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:45.080347 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.080312    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ebe4e56-jf4gh\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:45.080568 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.080369    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ebe4e56-jf4gh\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:45.080568 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.080508    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42d5w\" (UniqueName: \"kubernetes.io/projected/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-kube-api-access-42d5w\") pod \"maas-api-revoke-sub-6ebe4e56-jf4gh\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:45.081026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.081004    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ebe4e56-jf4gh\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:45.082748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.082729    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ebe4e56-jf4gh\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:45.143835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.143751    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42d5w\" (UniqueName: \"kubernetes.io/projected/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-kube-api-access-42d5w\") pod \"maas-api-revoke-sub-6ebe4e56-jf4gh\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:45.236031 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.235962    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:45.620650 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.620613    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"]
Oct 30 19:11:45.626631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.626600    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:11:45.679960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.679926    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-903a82\""
Oct 30 19:11:45.680460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.680430    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-903a82-dockercfg-f5x7p\""
Oct 30 19:11:45.684251 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.684223    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"]
Oct 30 19:11:45.767517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.759625    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e6a4ee-ad97-40b0-a608-82756a285d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n\" (UID: \"c9e6a4ee-ad97-40b0-a608-82756a285d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:11:45.767517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.759883    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m247\" (UniqueName: \"kubernetes.io/projected/c9e6a4ee-ad97-40b0-a608-82756a285d9d-kube-api-access-8m247\") pod \"payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n\" (UID: \"c9e6a4ee-ad97-40b0-a608-82756a285d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:11:45.799827 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:11:45.799776    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636edd24_8b5b_4e31_b8b1_fce5b6fd03c0.slice/crio-c49a6bcf57eaa5b3d4445a41a27879bdb1fa2d2409ca43cca92f921777dd585b WatchSource:0}: Error finding container c49a6bcf57eaa5b3d4445a41a27879bdb1fa2d2409ca43cca92f921777dd585b: Status 404 returned error can't find the container with id c49a6bcf57eaa5b3d4445a41a27879bdb1fa2d2409ca43cca92f921777dd585b
Oct 30 19:11:45.918577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.864744    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m247\" (UniqueName: \"kubernetes.io/projected/c9e6a4ee-ad97-40b0-a608-82756a285d9d-kube-api-access-8m247\") pod \"payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n\" (UID: \"c9e6a4ee-ad97-40b0-a608-82756a285d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:11:45.918577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.864961    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e6a4ee-ad97-40b0-a608-82756a285d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n\" (UID: \"c9e6a4ee-ad97-40b0-a608-82756a285d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:11:45.918577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.866119    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e6a4ee-ad97-40b0-a608-82756a285d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n\" (UID: \"c9e6a4ee-ad97-40b0-a608-82756a285d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:11:45.918577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.869106    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"]
Oct 30 19:11:45.918577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.918000    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"]
Oct 30 19:11:45.930542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.927945    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"]
Oct 30 19:11:45.930542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.928063    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:11:45.948529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.937255    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m247\" (UniqueName: \"kubernetes.io/projected/c9e6a4ee-ad97-40b0-a608-82756a285d9d-kube-api-access-8m247\") pod \"payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n\" (UID: \"c9e6a4ee-ad97-40b0-a608-82756a285d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:11:45.948529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:45.944765    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:11:46.070332 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.070274    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p96x2\" (UniqueName: \"kubernetes.io/projected/c747f06b-e604-4aa9-a122-5a969cec8306-kube-api-access-p96x2\") pod \"payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r\" (UID: \"c747f06b-e604-4aa9-a122-5a969cec8306\") " pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:11:46.070520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.070448    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c747f06b-e604-4aa9-a122-5a969cec8306-plugins-config-volume\") pod \"payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r\" (UID: \"c747f06b-e604-4aa9-a122-5a969cec8306\") " pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:11:46.173357 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.171939    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c747f06b-e604-4aa9-a122-5a969cec8306-plugins-config-volume\") pod \"payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r\" (UID: \"c747f06b-e604-4aa9-a122-5a969cec8306\") " pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:11:46.173357 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.172227    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96x2\" (UniqueName: \"kubernetes.io/projected/c747f06b-e604-4aa9-a122-5a969cec8306-kube-api-access-p96x2\") pod \"payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r\" (UID: \"c747f06b-e604-4aa9-a122-5a969cec8306\") " pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:11:46.173357 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.172880    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c747f06b-e604-4aa9-a122-5a969cec8306-plugins-config-volume\") pod \"payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r\" (UID: \"c747f06b-e604-4aa9-a122-5a969cec8306\") " pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:11:46.230987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.230949    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96x2\" (UniqueName: \"kubernetes.io/projected/c747f06b-e604-4aa9-a122-5a969cec8306-kube-api-access-p96x2\") pod \"payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r\" (UID: \"c747f06b-e604-4aa9-a122-5a969cec8306\") " pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:11:46.294269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.294238    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:11:46.391257 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.391226    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"]
Oct 30 19:11:46.394987 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:11:46.394119    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e6a4ee_ad97_40b0_a608_82756a285d9d.slice/crio-c3b516ec7e02ef8ce146cd702136b1f5f2ff9817cecbcabd41a8877664e57065 WatchSource:0}: Error finding container c3b516ec7e02ef8ce146cd702136b1f5f2ff9817cecbcabd41a8877664e57065: Status 404 returned error can't find the container with id c3b516ec7e02ef8ce146cd702136b1f5f2ff9817cecbcabd41a8877664e57065
Oct 30 19:11:46.565573 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.565528    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n" event={"ID":"c9e6a4ee-ad97-40b0-a608-82756a285d9d","Type":"ContainerStarted","Data":"39aadef5027ef5d5e2fe9b259c7af99759e265eede0fe1d76e2bd3fd7ea00ae0"}
Oct 30 19:11:46.565759 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.565640    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:11:46.565759 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.565666    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n" event={"ID":"c9e6a4ee-ad97-40b0-a608-82756a285d9d","Type":"ContainerStarted","Data":"c3b516ec7e02ef8ce146cd702136b1f5f2ff9817cecbcabd41a8877664e57065"}
Oct 30 19:11:46.567891 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.567861    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"]
Oct 30 19:11:46.572619 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.572593    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh" event={"ID":"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0","Type":"ContainerStarted","Data":"cfb669fe45a432447e456da34ba768b2a40bd795f5904b75a03f4892a953718b"}
Oct 30 19:11:46.572757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.572626    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh" event={"ID":"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0","Type":"ContainerStarted","Data":"c49a6bcf57eaa5b3d4445a41a27879bdb1fa2d2409ca43cca92f921777dd585b"}
Oct 30 19:11:46.572757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.572693    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:46.638042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.638005    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"]
Oct 30 19:11:46.680984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.680941    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:46.681234 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.681021    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:46.681234 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.681124    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjh4v\" (UniqueName: \"kubernetes.io/projected/e8875ee2-dc16-45e7-a136-72bb59a97eb0-kube-api-access-vjh4v\") pod \"maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:46.695659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.695572    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"]
Oct 30 19:11:46.700973 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.700931    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n" podStartSLOduration=1.700916784 podStartE2EDuration="1.700916784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:46.690421796 +0000 UTC m=+1997.578338886" watchObservedRunningTime="2026-10-30 19:11:46.700916784 +0000 UTC m=+1997.588833872"
Oct 30 19:11:46.704610 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:11:46.704581    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc747f06b_e604_4aa9_a122_5a969cec8306.slice/crio-f06d63b612a5521277d7c19d9ee9ec3f1f41a87067597ad7b2c5468a1e354125 WatchSource:0}: Error finding container f06d63b612a5521277d7c19d9ee9ec3f1f41a87067597ad7b2c5468a1e354125: Status 404 returned error can't find the container with id f06d63b612a5521277d7c19d9ee9ec3f1f41a87067597ad7b2c5468a1e354125
Oct 30 19:11:46.783275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.783244    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:46.783425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.783321    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjh4v\" (UniqueName: \"kubernetes.io/projected/e8875ee2-dc16-45e7-a136-72bb59a97eb0-kube-api-access-vjh4v\") pod \"maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:46.783654 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.783631    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:46.784060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.784039    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:46.786033 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.786012    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:46.837447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.837375    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh" podStartSLOduration=2.83735403 podStartE2EDuration="2.83735403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:46.815191153 +0000 UTC m=+1997.703108242" watchObservedRunningTime="2026-10-30 19:11:46.83735403 +0000 UTC m=+1997.725271120"
Oct 30 19:11:46.853201 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.850653    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjh4v\" (UniqueName: \"kubernetes.io/projected/e8875ee2-dc16-45e7-a136-72bb59a97eb0-kube-api-access-vjh4v\") pod \"maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:46.887658 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:46.887623    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:47.271970 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:11:47.271938    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8875ee2_dc16_45e7_a136_72bb59a97eb0.slice/crio-a88d68b7346e331a91ad367afbb2d75cbafd93a4cf0827496f36fdf2eb821a75 WatchSource:0}: Error finding container a88d68b7346e331a91ad367afbb2d75cbafd93a4cf0827496f36fdf2eb821a75: Status 404 returned error can't find the container with id a88d68b7346e331a91ad367afbb2d75cbafd93a4cf0827496f36fdf2eb821a75
Oct 30 19:11:47.273761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:47.273698    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"]
Oct 30 19:11:47.578711 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:47.578624    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r" event={"ID":"c747f06b-e604-4aa9-a122-5a969cec8306","Type":"ContainerStarted","Data":"d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089"}
Oct 30 19:11:47.578711 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:47.578674    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r" event={"ID":"c747f06b-e604-4aa9-a122-5a969cec8306","Type":"ContainerStarted","Data":"f06d63b612a5521277d7c19d9ee9ec3f1f41a87067597ad7b2c5468a1e354125"}
Oct 30 19:11:47.578937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:47.578862    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:11:47.580478 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:47.580451    4267 generic.go:356] "Generic (PLEG): container finished" podID="636edd24-8b5b-4e31-b8b1-fce5b6fd03c0" containerID="cfb669fe45a432447e456da34ba768b2a40bd795f5904b75a03f4892a953718b" exitCode=0
Oct 30 19:11:47.580637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:47.580540    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh" event={"ID":"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0","Type":"ContainerDied","Data":"cfb669fe45a432447e456da34ba768b2a40bd795f5904b75a03f4892a953718b"}
Oct 30 19:11:47.581952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:47.581931    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd" event={"ID":"e8875ee2-dc16-45e7-a136-72bb59a97eb0","Type":"ContainerStarted","Data":"a88d68b7346e331a91ad367afbb2d75cbafd93a4cf0827496f36fdf2eb821a75"}
Oct 30 19:11:47.697651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:47.697588    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r" podStartSLOduration=2.6975712 podStartE2EDuration="2.6975712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:47.692811785 +0000 UTC m=+1998.580728875" watchObservedRunningTime="2026-10-30 19:11:47.6975712 +0000 UTC m=+1998.585488289"
Oct 30 19:11:48.590836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:48.590742    4267 generic.go:356] "Generic (PLEG): container finished" podID="e8875ee2-dc16-45e7-a136-72bb59a97eb0" containerID="84e9853b268c8acc6b14234a982b51d0aca9574a9646569d39ed90c4ee248164" exitCode=0
Oct 30 19:11:48.591310 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:48.590827    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd" event={"ID":"e8875ee2-dc16-45e7-a136-72bb59a97eb0","Type":"ContainerDied","Data":"84e9853b268c8acc6b14234a982b51d0aca9574a9646569d39ed90c4ee248164"}
Oct 30 19:11:48.802888 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:48.802862    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:48.919073 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:48.918982    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-kube-api-access-42d5w\" (UniqueName: \"kubernetes.io/projected/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-kube-api-access-42d5w\") pod \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") "
Oct 30 19:11:48.919073 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:48.919026    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-openshift-ca\") pod \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") "
Oct 30 19:11:48.919391 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:48.919371    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-xks-ca\") pod \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\" (UID: \"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0\") "
Oct 30 19:11:48.919484 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:48.919464    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-openshift-ca" pod "636edd24-8b5b-4e31-b8b1-fce5b6fd03c0" (UID: "636edd24-8b5b-4e31-b8b1-fce5b6fd03c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:48.920021 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:48.919993    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:48.921475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:48.921448    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-xks-ca" pod "636edd24-8b5b-4e31-b8b1-fce5b6fd03c0" (UID: "636edd24-8b5b-4e31-b8b1-fce5b6fd03c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:48.921808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:48.921774    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-kube-api-access-42d5w" pod "636edd24-8b5b-4e31-b8b1-fce5b6fd03c0" (UID: "636edd24-8b5b-4e31-b8b1-fce5b6fd03c0"). InnerVolumeSpecName "kube-api-access-42d5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:49.021906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.021867    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:49.021906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.021899    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42d5w\" (UniqueName: \"kubernetes.io/projected/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0-kube-api-access-42d5w\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:49.599896 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.599805    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh" event={"ID":"636edd24-8b5b-4e31-b8b1-fce5b6fd03c0","Type":"ContainerDied","Data":"c49a6bcf57eaa5b3d4445a41a27879bdb1fa2d2409ca43cca92f921777dd585b"}
Oct 30 19:11:49.599896 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.599867    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c49a6bcf57eaa5b3d4445a41a27879bdb1fa2d2409ca43cca92f921777dd585b"
Oct 30 19:11:49.599896 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.599828    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"
Oct 30 19:11:49.764878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.764846    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:49.940430 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.940387    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-openshift-ca\") pod \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") "
Oct 30 19:11:49.940637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.940475    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-xks-ca\") pod \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") "
Oct 30 19:11:49.940637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.940620    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8875ee2-dc16-45e7-a136-72bb59a97eb0-kube-api-access-vjh4v\" (UniqueName: \"kubernetes.io/projected/e8875ee2-dc16-45e7-a136-72bb59a97eb0-kube-api-access-vjh4v\") pod \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\" (UID: \"e8875ee2-dc16-45e7-a136-72bb59a97eb0\") "
Oct 30 19:11:49.941128 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.940858    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-openshift-ca" pod "e8875ee2-dc16-45e7-a136-72bb59a97eb0" (UID: "e8875ee2-dc16-45e7-a136-72bb59a97eb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:49.941617 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.941591    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:49.944138 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.944091    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-xks-ca" pod "e8875ee2-dc16-45e7-a136-72bb59a97eb0" (UID: "e8875ee2-dc16-45e7-a136-72bb59a97eb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:49.944901 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:49.944861    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8875ee2-dc16-45e7-a136-72bb59a97eb0-kube-api-access-vjh4v" pod "e8875ee2-dc16-45e7-a136-72bb59a97eb0" (UID: "e8875ee2-dc16-45e7-a136-72bb59a97eb0"). InnerVolumeSpecName "kube-api-access-vjh4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:50.042952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:50.042917    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8875ee2-dc16-45e7-a136-72bb59a97eb0-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:50.042952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:50.042950    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjh4v\" (UniqueName: \"kubernetes.io/projected/e8875ee2-dc16-45e7-a136-72bb59a97eb0-kube-api-access-vjh4v\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:50.162422 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:11:50.162387    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:11:50.612220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:50.612185    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"
Oct 30 19:11:50.612220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:50.612220    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd" event={"ID":"e8875ee2-dc16-45e7-a136-72bb59a97eb0","Type":"ContainerDied","Data":"a88d68b7346e331a91ad367afbb2d75cbafd93a4cf0827496f36fdf2eb821a75"}
Oct 30 19:11:50.612766 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:50.612282    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a88d68b7346e331a91ad367afbb2d75cbafd93a4cf0827496f36fdf2eb821a75"
Oct 30 19:11:52.362859 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:52.362823    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq" podUID="482f2d86-6da5-48b0-a8ff-52f3db5be0f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:11:53.929444 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:53.929403    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"]
Oct 30 19:11:53.930010 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:53.929694    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy" containerID="cri-o://27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30" gracePeriod=30
Oct 30 19:11:55.846645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.846600    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"]
Oct 30 19:11:55.847407 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.847387    4267 cpu_manager.go:414] "containerMap: removing container" podUID="636edd24-8b5b-4e31-b8b1-fce5b6fd03c0" containerName="revoke-keys"
Oct 30 19:11:55.847407 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.847408    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636edd24-8b5b-4e31-b8b1-fce5b6fd03c0" containerName="revoke-keys"
Oct 30 19:11:55.847601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.847459    4267 cpu_manager.go:414] "containerMap: removing container" podUID="e8875ee2-dc16-45e7-a136-72bb59a97eb0" containerName="revoke-keys"
Oct 30 19:11:55.847601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.847467    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8875ee2-dc16-45e7-a136-72bb59a97eb0" containerName="revoke-keys"
Oct 30 19:11:55.847671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.847622    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8875ee2-dc16-45e7-a136-72bb59a97eb0" containerName="revoke-keys"
Oct 30 19:11:55.847671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.847638    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="636edd24-8b5b-4e31-b8b1-fce5b6fd03c0" containerName="revoke-keys"
Oct 30 19:11:55.851365 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.851349    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:55.903856 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.903817    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-322e97-serving-cert\""
Oct 30 19:11:55.938042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.937990    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-metrics-certs\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:55.938246 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.938158    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-openshift-service-ca\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:55.938384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.938359    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtqjg\" (UniqueName: \"kubernetes.io/projected/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-kube-api-access-qtqjg\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:55.938476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.938430    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-maas-api-tls\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:55.960879 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:55.960840    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"]
Oct 30 19:11:56.040667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.040624    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-metrics-certs\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:56.040878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.040694    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-openshift-service-ca\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:56.040878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.040812    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtqjg\" (UniqueName: \"kubernetes.io/projected/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-kube-api-access-qtqjg\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:56.040878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.040848    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-maas-api-tls\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:56.041054 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:11:56.041033    4267 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-serving-cert: secret "maas-api-e2e-inf-a-322e97-serving-cert" not found
Oct 30 19:11:56.041146 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:11:56.041119    4267 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-maas-api-tls podName:8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330 nodeName:}" failed. No retries permitted until 2026-10-30 19:11:56.541095224 +0000 UTC m=+2007.429012291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-maas-api-tls") pod "maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz" (UID: "8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330") : secret "maas-api-e2e-inf-a-322e97-serving-cert" not found
Oct 30 19:11:56.041530 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.041473    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-openshift-service-ca\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:56.043237 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.043211    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-metrics-certs\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:56.146686 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.146605    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtqjg\" (UniqueName: \"kubernetes.io/projected/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-kube-api-access-qtqjg\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:56.368711 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.368675    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:11:56.368973 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.368942    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:11:56.556165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.556121    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-maas-api-tls\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:56.558819 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.558792    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-maas-api-tls\") pod \"maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:56.765543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:56.765479    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:57.154598 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:11:57.154569    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8000ef1b_4de0_4a3a_ba78_a4f5aaa6c330.slice/crio-ff17a6d203f6523544a5b05b4518d6cb5f92b34163b50fb178b373c7c7b4fd73 WatchSource:0}: Error finding container ff17a6d203f6523544a5b05b4518d6cb5f92b34163b50fb178b373c7c7b4fd73: Status 404 returned error can't find the container with id ff17a6d203f6523544a5b05b4518d6cb5f92b34163b50fb178b373c7c7b4fd73
Oct 30 19:11:57.156461 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:57.156439    4267 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:11:57.156999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:57.156973    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"]
Oct 30 19:11:57.585344 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:57.585310    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:11:57.680175 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:57.680129    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz" event={"ID":"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330","Type":"ContainerStarted","Data":"ff17a6d203f6523544a5b05b4518d6cb5f92b34163b50fb178b373c7c7b4fd73"}
Oct 30 19:11:57.754077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:57.754045    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"]
Oct 30 19:11:57.851183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:57.851054    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"]
Oct 30 19:11:58.594033 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:58.594003    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:11:58.692456 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:58.692416    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz" event={"ID":"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330","Type":"ContainerStarted","Data":"a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3"}
Oct 30 19:11:58.692663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:58.692480    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:11:58.829704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:58.829643    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz" podStartSLOduration=2.9908271429999997 podStartE2EDuration="3.829620446s" totalImagesPullingTime="838.793303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:58.82744209 +0000 UTC m=+2009.715359180" watchObservedRunningTime="2026-10-30 19:11:58.829620446 +0000 UTC m=+2009.717537530"
Oct 30 19:11:59.447741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.447709    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 19:11:59.535090 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535055    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-podinfo\") pod \"3de3ab02-b017-4de5-87c0-d102e55164fd\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") "
Oct 30 19:11:59.535292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535121    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-certs\") pod \"3de3ab02-b017-4de5-87c0-d102e55164fd\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") "
Oct 30 19:11:59.535292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535196    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de3ab02-b017-4de5-87c0-d102e55164fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de3ab02-b017-4de5-87c0-d102e55164fd-istiod-ca-cert\") pod \"3de3ab02-b017-4de5-87c0-d102e55164fd\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") "
Oct 30 19:11:59.535292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535255    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-credential-socket\") pod \"3de3ab02-b017-4de5-87c0-d102e55164fd\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") "
Oct 30 19:11:59.535482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535317    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-data\") pod \"3de3ab02-b017-4de5-87c0-d102e55164fd\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") "
Oct 30 19:11:59.535482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535335    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-envoy\") pod \"3de3ab02-b017-4de5-87c0-d102e55164fd\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") "
Oct 30 19:11:59.535482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535454    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-kube-api-access-ql84n\" (UniqueName: \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-kube-api-access-ql84n\") pod \"3de3ab02-b017-4de5-87c0-d102e55164fd\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") "
Oct 30 19:11:59.535671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535453    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-certs" pod "3de3ab02-b017-4de5-87c0-d102e55164fd" (UID: "3de3ab02-b017-4de5-87c0-d102e55164fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:59.535671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535581    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-socket\") pod \"3de3ab02-b017-4de5-87c0-d102e55164fd\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") "
Oct 30 19:11:59.535671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535612    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-token\" (UniqueName: \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-token\") pod \"3de3ab02-b017-4de5-87c0-d102e55164fd\" (UID: \"3de3ab02-b017-4de5-87c0-d102e55164fd\") "
Oct 30 19:11:59.535671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535636    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-data" pod "3de3ab02-b017-4de5-87c0-d102e55164fd" (UID: "3de3ab02-b017-4de5-87c0-d102e55164fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:59.535891 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535691    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de3ab02-b017-4de5-87c0-d102e55164fd-istiod-ca-cert" pod "3de3ab02-b017-4de5-87c0-d102e55164fd" (UID: "3de3ab02-b017-4de5-87c0-d102e55164fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:59.535891 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535854    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-socket" pod "3de3ab02-b017-4de5-87c0-d102e55164fd" (UID: "3de3ab02-b017-4de5-87c0-d102e55164fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:59.536001 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.535987    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-credential-socket" pod "3de3ab02-b017-4de5-87c0-d102e55164fd" (UID: "3de3ab02-b017-4de5-87c0-d102e55164fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:59.536409 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.536385    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de3ab02-b017-4de5-87c0-d102e55164fd-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:59.536409 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.536407    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:59.536630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.536416    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:59.536630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.536427    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:59.536630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.536435    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:59.537915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.537884    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-kube-api-access-ql84n" pod "3de3ab02-b017-4de5-87c0-d102e55164fd" (UID: "3de3ab02-b017-4de5-87c0-d102e55164fd"). InnerVolumeSpecName "kube-api-access-ql84n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:59.538082 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.538066    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-podinfo" pod "3de3ab02-b017-4de5-87c0-d102e55164fd" (UID: "3de3ab02-b017-4de5-87c0-d102e55164fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:11:59.538171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.538155    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-envoy" pod "3de3ab02-b017-4de5-87c0-d102e55164fd" (UID: "3de3ab02-b017-4de5-87c0-d102e55164fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:59.538210 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.538176    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-token" pod "3de3ab02-b017-4de5-87c0-d102e55164fd" (UID: "3de3ab02-b017-4de5-87c0-d102e55164fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:59.645361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.643130    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:59.645361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.643162    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:59.645361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.643171    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql84n\" (UniqueName: \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-kube-api-access-ql84n\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:59.645361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.643182    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de3ab02-b017-4de5-87c0-d102e55164fd-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:11:59.707577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.707480    4267 generic.go:356] "Generic (PLEG): container finished" podID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerID="27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30" exitCode=0
Oct 30 19:11:59.708016 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.707989    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"
Oct 30 19:11:59.708163 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.708086    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" event={"ID":"3de3ab02-b017-4de5-87c0-d102e55164fd","Type":"ContainerDied","Data":"27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30"}
Oct 30 19:11:59.708233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.708173    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx" event={"ID":"3de3ab02-b017-4de5-87c0-d102e55164fd","Type":"ContainerDied","Data":"3c7abe5d1dbdf6e74f9e6c1d80188979009755f28f06ccff98a1ee668e6295dd"}
Oct 30 19:11:59.708233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.708191    4267 scope.go:122] "RemoveContainer" containerID="27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30"
Oct 30 19:11:59.733256 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.733227    4267 scope.go:122] "RemoveContainer" containerID="27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30"
Oct 30 19:11:59.733635 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:11:59.733608    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30\": container with ID starting with 27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30 not found: ID does not exist" containerID="27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30"
Oct 30 19:11:59.733753 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.733644    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30"} err="failed to get container status \"27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30\": rpc error: code = NotFound desc = could not find container \"27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30\": container with ID starting with 27afde29ca192032d5873ade0029b76907eee05cd52948167e5031f40df3bb30 not found: ID does not exist"
Oct 30 19:11:59.880032 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.879886    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"]
Oct 30 19:11:59.884759 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:11:59.884726    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-gg4xx"]
Oct 30 19:12:01.224322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:01.224289    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" path="/var/lib/kubelet/pods/3de3ab02-b017-4de5-87c0-d102e55164fd/volumes"
Oct 30 19:12:02.363209 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:02.363164    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq" podUID="482f2d86-6da5-48b0-a8ff-52f3db5be0f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:12:02.754467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:02.754414    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck" podUID="93d1eca9-8bcc-42d2-a86e-9719996dbb22" containerName="payload-processing" containerID="cri-o://0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc" gracePeriod=25
Oct 30 19:12:02.852429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:02.852374    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk" podUID="2226498d-e1e2-446d-a802-a1b107bd5cf5" containerName="payload-pre-processing" containerID="cri-o://0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31" gracePeriod=25
Oct 30 19:12:03.086802 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.086620    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:12:03.198119 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.198083    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d1eca9-8bcc-42d2-a86e-9719996dbb22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d1eca9-8bcc-42d2-a86e-9719996dbb22-plugins-config-volume\") pod \"93d1eca9-8bcc-42d2-a86e-9719996dbb22\" (UID: \"93d1eca9-8bcc-42d2-a86e-9719996dbb22\") "
Oct 30 19:12:03.198248 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.198160    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d1eca9-8bcc-42d2-a86e-9719996dbb22-kube-api-access-r4rkw\" (UniqueName: \"kubernetes.io/projected/93d1eca9-8bcc-42d2-a86e-9719996dbb22-kube-api-access-r4rkw\") pod \"93d1eca9-8bcc-42d2-a86e-9719996dbb22\" (UID: \"93d1eca9-8bcc-42d2-a86e-9719996dbb22\") "
Oct 30 19:12:03.199037 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.199005    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d1eca9-8bcc-42d2-a86e-9719996dbb22-plugins-config-volume" pod "93d1eca9-8bcc-42d2-a86e-9719996dbb22" (UID: "93d1eca9-8bcc-42d2-a86e-9719996dbb22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:03.199750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.199722    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d1eca9-8bcc-42d2-a86e-9719996dbb22-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:03.243095 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.243045    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d1eca9-8bcc-42d2-a86e-9719996dbb22-kube-api-access-r4rkw" pod "93d1eca9-8bcc-42d2-a86e-9719996dbb22" (UID: "93d1eca9-8bcc-42d2-a86e-9719996dbb22"). InnerVolumeSpecName "kube-api-access-r4rkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:03.256429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.255562    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:12:03.302468 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.302201    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4rkw\" (UniqueName: \"kubernetes.io/projected/93d1eca9-8bcc-42d2-a86e-9719996dbb22-kube-api-access-r4rkw\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:03.403014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.402976    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2226498d-e1e2-446d-a802-a1b107bd5cf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2226498d-e1e2-446d-a802-a1b107bd5cf5-plugins-config-volume\") pod \"2226498d-e1e2-446d-a802-a1b107bd5cf5\" (UID: \"2226498d-e1e2-446d-a802-a1b107bd5cf5\") "
Oct 30 19:12:03.403561 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.403213    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2226498d-e1e2-446d-a802-a1b107bd5cf5-kube-api-access-rmwpr\" (UniqueName: \"kubernetes.io/projected/2226498d-e1e2-446d-a802-a1b107bd5cf5-kube-api-access-rmwpr\") pod \"2226498d-e1e2-446d-a802-a1b107bd5cf5\" (UID: \"2226498d-e1e2-446d-a802-a1b107bd5cf5\") "
Oct 30 19:12:03.404720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.404688    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2226498d-e1e2-446d-a802-a1b107bd5cf5-plugins-config-volume" pod "2226498d-e1e2-446d-a802-a1b107bd5cf5" (UID: "2226498d-e1e2-446d-a802-a1b107bd5cf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:03.406972 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.406938    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2226498d-e1e2-446d-a802-a1b107bd5cf5-kube-api-access-rmwpr" pod "2226498d-e1e2-446d-a802-a1b107bd5cf5" (UID: "2226498d-e1e2-446d-a802-a1b107bd5cf5"). InnerVolumeSpecName "kube-api-access-rmwpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:03.505686 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.505643    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmwpr\" (UniqueName: \"kubernetes.io/projected/2226498d-e1e2-446d-a802-a1b107bd5cf5-kube-api-access-rmwpr\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:03.505867 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.505683    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2226498d-e1e2-446d-a802-a1b107bd5cf5-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:03.747254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.747214    4267 generic.go:356] "Generic (PLEG): container finished" podID="93d1eca9-8bcc-42d2-a86e-9719996dbb22" containerID="0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc" exitCode=0
Oct 30 19:12:03.747443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.747298    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"
Oct 30 19:12:03.747443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.747387    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck" event={"ID":"93d1eca9-8bcc-42d2-a86e-9719996dbb22","Type":"ContainerDied","Data":"0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc"}
Oct 30 19:12:03.747443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.747436    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck" event={"ID":"93d1eca9-8bcc-42d2-a86e-9719996dbb22","Type":"ContainerDied","Data":"194cc35a288d7ca7a856291d57677207bb69d8c039d234df200935690a9aaaaa"}
Oct 30 19:12:03.747927 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.747471    4267 scope.go:122] "RemoveContainer" containerID="0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc"
Oct 30 19:12:03.750064 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.750037    4267 generic.go:356] "Generic (PLEG): container finished" podID="2226498d-e1e2-446d-a802-a1b107bd5cf5" containerID="0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31" exitCode=0
Oct 30 19:12:03.750260 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.750243    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk" event={"ID":"2226498d-e1e2-446d-a802-a1b107bd5cf5","Type":"ContainerDied","Data":"0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31"}
Oct 30 19:12:03.750332 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.750286    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk" event={"ID":"2226498d-e1e2-446d-a802-a1b107bd5cf5","Type":"ContainerDied","Data":"27c2daf1440d0cfbdc9695fc2dc1b76e8ef386285ee35fe0fb766b4cdc810693"}
Oct 30 19:12:03.750388 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.750345    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"
Oct 30 19:12:03.790613 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.790586    4267 scope.go:122] "RemoveContainer" containerID="0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc"
Oct 30 19:12:03.790978 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:12:03.790945    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc\": container with ID starting with 0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc not found: ID does not exist" containerID="0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc"
Oct 30 19:12:03.790978 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.790984    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc"} err="failed to get container status \"0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc\": rpc error: code = NotFound desc = could not find container \"0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc\": container with ID starting with 0e7e3d7ca3d5f3bdab15f0d8a27633291ecfade1806d02bdca36ffec142107dc not found: ID does not exist"
Oct 30 19:12:03.791179 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.791004    4267 scope.go:122] "RemoveContainer" containerID="0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31"
Oct 30 19:12:03.804135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.804017    4267 scope.go:122] "RemoveContainer" containerID="0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31"
Oct 30 19:12:03.804396 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:12:03.804366    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31\": container with ID starting with 0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31 not found: ID does not exist" containerID="0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31"
Oct 30 19:12:03.804479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.804406    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31"} err="failed to get container status \"0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31\": rpc error: code = NotFound desc = could not find container \"0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31\": container with ID starting with 0eb0778cfde95c5549c64098c27a7f6bb4cce44e8847439e10e675790c0edc31 not found: ID does not exist"
Oct 30 19:12:03.920972 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.920877    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"]
Oct 30 19:12:03.967575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:03.967545    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b8d0f-67b8758555-h5srk"]
Oct 30 19:12:04.011292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.011260    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:12:04.077154 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.077111    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"]
Oct 30 19:12:04.081810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.081781    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b8d0f-75cfc444f9-qhjck"]
Oct 30 19:12:04.114884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.114849    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-plugins-config-volume\") pod \"482f2d86-6da5-48b0-a8ff-52f3db5be0f1\" (UID: \"482f2d86-6da5-48b0-a8ff-52f3db5be0f1\") "
Oct 30 19:12:04.115082 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.115052    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-kube-api-access-rt7fj\" (UniqueName: \"kubernetes.io/projected/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-kube-api-access-rt7fj\") pod \"482f2d86-6da5-48b0-a8ff-52f3db5be0f1\" (UID: \"482f2d86-6da5-48b0-a8ff-52f3db5be0f1\") "
Oct 30 19:12:04.115290 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.115250    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-plugins-config-volume" pod "482f2d86-6da5-48b0-a8ff-52f3db5be0f1" (UID: "482f2d86-6da5-48b0-a8ff-52f3db5be0f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:04.115859 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.115834    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:04.117320 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.117297    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-kube-api-access-rt7fj" pod "482f2d86-6da5-48b0-a8ff-52f3db5be0f1" (UID: "482f2d86-6da5-48b0-a8ff-52f3db5be0f1"). InnerVolumeSpecName "kube-api-access-rt7fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:04.217478 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.217441    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt7fj\" (UniqueName: \"kubernetes.io/projected/482f2d86-6da5-48b0-a8ff-52f3db5be0f1-kube-api-access-rt7fj\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:04.713487 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.713455    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:12:04.762060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.762026    4267 generic.go:356] "Generic (PLEG): container finished" podID="482f2d86-6da5-48b0-a8ff-52f3db5be0f1" containerID="c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775" exitCode=0
Oct 30 19:12:04.762220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.762078    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"
Oct 30 19:12:04.762220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.762114    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq" event={"ID":"482f2d86-6da5-48b0-a8ff-52f3db5be0f1","Type":"ContainerDied","Data":"c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775"}
Oct 30 19:12:04.762220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.762168    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq" event={"ID":"482f2d86-6da5-48b0-a8ff-52f3db5be0f1","Type":"ContainerDied","Data":"e385e1ccb01e3ef1326a322bcc257a24945953756e5b4418f2c15e7c390d6e91"}
Oct 30 19:12:04.762220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.762195    4267 scope.go:122] "RemoveContainer" containerID="c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775"
Oct 30 19:12:04.773429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.773411    4267 scope.go:122] "RemoveContainer" containerID="c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775"
Oct 30 19:12:04.773719 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:12:04.773699    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775\": container with ID starting with c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775 not found: ID does not exist" containerID="c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775"
Oct 30 19:12:04.773793 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.773726    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775"} err="failed to get container status \"c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775\": rpc error: code = NotFound desc = could not find container \"c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775\": container with ID starting with c20c18ee8c82ce61a1754925cf72218aa05cb7077084ec548193e4fad034d775 not found: ID does not exist"
Oct 30 19:12:04.969573 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.965920    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"]
Oct 30 19:12:04.972663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:04.972636    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae6a138d-75fd686cbf-zj7xq"]
Oct 30 19:12:05.225886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:05.225790    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2226498d-e1e2-446d-a802-a1b107bd5cf5" path="/var/lib/kubelet/pods/2226498d-e1e2-446d-a802-a1b107bd5cf5/volumes"
Oct 30 19:12:05.226527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:05.226479    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482f2d86-6da5-48b0-a8ff-52f3db5be0f1" path="/var/lib/kubelet/pods/482f2d86-6da5-48b0-a8ff-52f3db5be0f1/volumes"
Oct 30 19:12:05.227093 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:05.227073    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d1eca9-8bcc-42d2-a86e-9719996dbb22" path="/var/lib/kubelet/pods/93d1eca9-8bcc-42d2-a86e-9719996dbb22/volumes"
Oct 30 19:12:06.076594 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.076560    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"]
Oct 30 19:12:06.077266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077251    4267 cpu_manager.go:414] "containerMap: removing container" podUID="93d1eca9-8bcc-42d2-a86e-9719996dbb22" containerName="payload-processing"
Oct 30 19:12:06.077308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077266    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d1eca9-8bcc-42d2-a86e-9719996dbb22" containerName="payload-processing"
Oct 30 19:12:06.077308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077287    4267 cpu_manager.go:414] "containerMap: removing container" podUID="2226498d-e1e2-446d-a802-a1b107bd5cf5" containerName="payload-pre-processing"
Oct 30 19:12:06.077308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077292    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2226498d-e1e2-446d-a802-a1b107bd5cf5" containerName="payload-pre-processing"
Oct 30 19:12:06.077399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077324    4267 cpu_manager.go:414] "containerMap: removing container" podUID="482f2d86-6da5-48b0-a8ff-52f3db5be0f1" containerName="payload-processing"
Oct 30 19:12:06.077399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077330    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482f2d86-6da5-48b0-a8ff-52f3db5be0f1" containerName="payload-processing"
Oct 30 19:12:06.077399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077347    4267 cpu_manager.go:414] "containerMap: removing container" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy"
Oct 30 19:12:06.077399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077352    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy"
Oct 30 19:12:06.077570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077448    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d1eca9-8bcc-42d2-a86e-9719996dbb22" containerName="payload-processing"
Oct 30 19:12:06.077570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077459    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="2226498d-e1e2-446d-a802-a1b107bd5cf5" containerName="payload-pre-processing"
Oct 30 19:12:06.077570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077470    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="482f2d86-6da5-48b0-a8ff-52f3db5be0f1" containerName="payload-processing"
Oct 30 19:12:06.077570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.077478    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de3ab02-b017-4de5-87c0-d102e55164fd" containerName="istio-proxy"
Oct 30 19:12:06.082456 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.082434    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.137866 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.137836    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-24e88a-serving-cert\""
Oct 30 19:12:06.194174 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.194133    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"]
Oct 30 19:12:06.248223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.248187    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-maas-api-tls\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.248458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.248256    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbb4x\" (UniqueName: \"kubernetes.io/projected/078a0cc5-96c9-49b4-bd07-91cc514a556e-kube-api-access-vbb4x\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.248458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.248306    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078a0cc5-96c9-49b4-bd07-91cc514a556e-openshift-service-ca\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.248458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.248348    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-metrics-certs\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.351016 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.350917    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-maas-api-tls\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.351363 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.351330    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbb4x\" (UniqueName: \"kubernetes.io/projected/078a0cc5-96c9-49b4-bd07-91cc514a556e-kube-api-access-vbb4x\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.353198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.353174    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078a0cc5-96c9-49b4-bd07-91cc514a556e-openshift-service-ca\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.359211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.359183    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-metrics-certs\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.370579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.357374    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078a0cc5-96c9-49b4-bd07-91cc514a556e-openshift-service-ca\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.370579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.369156    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-maas-api-tls\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.370579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.369317    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-metrics-certs\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.410631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.410594    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbb4x\" (UniqueName: \"kubernetes.io/projected/078a0cc5-96c9-49b4-bd07-91cc514a556e-kube-api-access-vbb4x\") pod \"maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:06.697386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:06.697311    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:07.083904 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:12:07.083874    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078a0cc5_96c9_49b4_bd07_91cc514a556e.slice/crio-31d958dfcb2b10e8801b2d49321644c847845e307d8ee916158638de761d4e56 WatchSource:0}: Error finding container 31d958dfcb2b10e8801b2d49321644c847845e307d8ee916158638de761d4e56: Status 404 returned error can't find the container with id 31d958dfcb2b10e8801b2d49321644c847845e307d8ee916158638de761d4e56
Oct 30 19:12:07.086484 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:07.086460    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"]
Oct 30 19:12:07.811073 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:07.811036    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q" event={"ID":"078a0cc5-96c9-49b4-bd07-91cc514a556e","Type":"ContainerStarted","Data":"31d958dfcb2b10e8801b2d49321644c847845e307d8ee916158638de761d4e56"}
Oct 30 19:12:08.821191 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:08.821148    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q" event={"ID":"078a0cc5-96c9-49b4-bd07-91cc514a556e","Type":"ContainerStarted","Data":"e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c"}
Oct 30 19:12:08.821616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:08.821284    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:08.938213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:08.938154    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q" podStartSLOduration=2.077808948 podStartE2EDuration="2.938135353s" totalImagesPullingTime="860.326405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:08.935545222 +0000 UTC m=+2019.823462309" watchObservedRunningTime="2026-10-30 19:12:08.938135353 +0000 UTC m=+2019.826052445"
Oct 30 19:12:14.836567 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:14.836471    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:12:20.159252 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:12:20.159211    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:12:32.540445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.540391    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"]
Oct 30 19:12:32.552608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.552341    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:32.589117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.589080    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"]
Oct 30 19:12:32.622682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.622611    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:32.623382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.623353    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz68m\" (UniqueName: \"kubernetes.io/projected/958a790f-dd68-46f5-ae31-7f31a976ba2d-kube-api-access-sz68m\") pod \"maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:32.624765 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.624726    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:32.633777 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.633752    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:12:32.634054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.634035    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:12:32.728960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.728920    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz68m\" (UniqueName: \"kubernetes.io/projected/958a790f-dd68-46f5-ae31-7f31a976ba2d-kube-api-access-sz68m\") pod \"maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:32.734552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.733731    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:32.734552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.733863    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:32.735033 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.735009    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:32.736784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.736759    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:32.786963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.786931    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz68m\" (UniqueName: \"kubernetes.io/projected/958a790f-dd68-46f5-ae31-7f31a976ba2d-kube-api-access-sz68m\") pod \"maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:32.871694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:32.871608    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:33.292981 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:12:33.292935    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958a790f_dd68_46f5_ae31_7f31a976ba2d.slice/crio-7d63500971f5fd4f5cf6b606e699e6579e81cfca2d1484804a95ca5ce96b2d65 WatchSource:0}: Error finding container 7d63500971f5fd4f5cf6b606e699e6579e81cfca2d1484804a95ca5ce96b2d65: Status 404 returned error can't find the container with id 7d63500971f5fd4f5cf6b606e699e6579e81cfca2d1484804a95ca5ce96b2d65
Oct 30 19:12:33.293310 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:33.293283    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"]
Oct 30 19:12:34.080803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:34.080765    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r" event={"ID":"958a790f-dd68-46f5-ae31-7f31a976ba2d","Type":"ContainerStarted","Data":"032a352a2235eb29106c69392901a1550e27078c18ca8261f358462ec9924770"}
Oct 30 19:12:34.081320 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:34.080821    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r" event={"ID":"958a790f-dd68-46f5-ae31-7f31a976ba2d","Type":"ContainerStarted","Data":"7d63500971f5fd4f5cf6b606e699e6579e81cfca2d1484804a95ca5ce96b2d65"}
Oct 30 19:12:34.198306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:34.198253    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r" podStartSLOduration=2.198239375 podStartE2EDuration="2.198239375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:34.195300616 +0000 UTC m=+2045.083217706" watchObservedRunningTime="2026-10-30 19:12:34.198239375 +0000 UTC m=+2045.086156485"
Oct 30 19:12:35.090726 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:35.090681    4267 generic.go:356] "Generic (PLEG): container finished" podID="958a790f-dd68-46f5-ae31-7f31a976ba2d" containerID="032a352a2235eb29106c69392901a1550e27078c18ca8261f358462ec9924770" exitCode=0
Oct 30 19:12:35.091141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:35.090771    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r" event={"ID":"958a790f-dd68-46f5-ae31-7f31a976ba2d","Type":"ContainerDied","Data":"032a352a2235eb29106c69392901a1550e27078c18ca8261f358462ec9924770"}
Oct 30 19:12:36.324957 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.324917    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:36.504148 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.504105    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958a790f-dd68-46f5-ae31-7f31a976ba2d-kube-api-access-sz68m\" (UniqueName: \"kubernetes.io/projected/958a790f-dd68-46f5-ae31-7f31a976ba2d-kube-api-access-sz68m\") pod \"958a790f-dd68-46f5-ae31-7f31a976ba2d\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") "
Oct 30 19:12:36.504331 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.504218    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-openshift-ca\") pod \"958a790f-dd68-46f5-ae31-7f31a976ba2d\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") "
Oct 30 19:12:36.504462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.504445    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-xks-ca\") pod \"958a790f-dd68-46f5-ae31-7f31a976ba2d\" (UID: \"958a790f-dd68-46f5-ae31-7f31a976ba2d\") "
Oct 30 19:12:36.504603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.504580    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-openshift-ca" pod "958a790f-dd68-46f5-ae31-7f31a976ba2d" (UID: "958a790f-dd68-46f5-ae31-7f31a976ba2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:36.505024 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.505002    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:36.506521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.506470    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-xks-ca" pod "958a790f-dd68-46f5-ae31-7f31a976ba2d" (UID: "958a790f-dd68-46f5-ae31-7f31a976ba2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:36.506521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.506510    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958a790f-dd68-46f5-ae31-7f31a976ba2d-kube-api-access-sz68m" pod "958a790f-dd68-46f5-ae31-7f31a976ba2d" (UID: "958a790f-dd68-46f5-ae31-7f31a976ba2d"). InnerVolumeSpecName "kube-api-access-sz68m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:36.607017 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.606968    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/958a790f-dd68-46f5-ae31-7f31a976ba2d-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:36.607017 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.607009    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz68m\" (UniqueName: \"kubernetes.io/projected/958a790f-dd68-46f5-ae31-7f31a976ba2d-kube-api-access-sz68m\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:12:36.823746 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.823651    4267 patch_prober.go:29] interesting pod/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:12:36.823901 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:36.823733    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:12:37.114120 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:37.114031    4267 generic.go:356] "Generic (PLEG): container finished" podID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerID="aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75" exitCode=0
Oct 30 19:12:37.114329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:37.114128    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" event={"ID":"60ce3cd7-8197-4758-bfc1-cff2f15d12ae","Type":"ContainerDied","Data":"aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75"}
Oct 30 19:12:37.114836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:37.114814    4267 scope.go:122] "RemoveContainer" containerID="aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75"
Oct 30 19:12:37.115925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:37.115801    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"
Oct 30 19:12:37.115925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:37.115846    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r" event={"ID":"958a790f-dd68-46f5-ae31-7f31a976ba2d","Type":"ContainerDied","Data":"7d63500971f5fd4f5cf6b606e699e6579e81cfca2d1484804a95ca5ce96b2d65"}
Oct 30 19:12:37.115925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:37.115891    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d63500971f5fd4f5cf6b606e699e6579e81cfca2d1484804a95ca5ce96b2d65"
Oct 30 19:12:37.116740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:37.116708    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:12:37.116821 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:37.116781    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:12:37.116821 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:37.116816    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:12:38.148339 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:38.148304    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" event={"ID":"60ce3cd7-8197-4758-bfc1-cff2f15d12ae","Type":"ContainerStarted","Data":"4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72"}
Oct 30 19:12:38.387980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:38.387946    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:12:38.388306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:38.388280    4267 patch_prober.go:29] interesting pod/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:12:38.388423 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:38.388333    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 19:12:39.388117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:39.388072    4267 patch_prober.go:29] interesting pod/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:12:39.388582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:39.388143    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 19:12:40.388834 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:40.388799    4267 patch_prober.go:29] interesting pod/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:12:40.389273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:40.388855    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 19:12:41.388156 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:41.388121    4267 patch_prober.go:29] interesting pod/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:12:41.388346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:41.388180    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 19:12:42.388077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:42.388039    4267 patch_prober.go:29] interesting pod/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:12:42.388531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:42.388102    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 19:12:43.388272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:43.388230    4267 patch_prober.go:29] interesting pod/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:12:43.388754 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:43.388327    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 19:12:44.392445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:44.392417    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:12:44.392924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:44.392910    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:12:44.393560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:44.393542    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:12:45.203012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.202977    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"]
Oct 30 19:12:45.203705 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.203690    4267 cpu_manager.go:414] "containerMap: removing container" podUID="958a790f-dd68-46f5-ae31-7f31a976ba2d" containerName="revoke-keys"
Oct 30 19:12:45.203763 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.203706    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958a790f-dd68-46f5-ae31-7f31a976ba2d" containerName="revoke-keys"
Oct 30 19:12:45.203855 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.203845    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="958a790f-dd68-46f5-ae31-7f31a976ba2d" containerName="revoke-keys"
Oct 30 19:12:45.206500 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.206474    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.298515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.284444    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-token\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.326148 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.325815    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-podinfo\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.326148 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.325920    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-socket\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.326148 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.326103    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-certs\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.326462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.326375    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tbds\" (UniqueName: \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-kube-api-access-6tbds\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.326603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.326562    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-data\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.326673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.326602    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-credential-socket\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.326673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.326643    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-envoy\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.326785 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.326685    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istiod-ca-cert\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.368542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.367949    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8da6e2-openshift-default-dockercfg-vrhnr\""
Oct 30 19:12:45.447302 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.447261    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-envoy\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.448406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.448380    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istiod-ca-cert\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.451065 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.451044    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-token\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.452566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.452541    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-podinfo\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.453225 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.453175    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-socket\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.453815 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.453799    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-certs\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.454296 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.454278    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tbds\" (UniqueName: \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-kube-api-access-6tbds\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.455211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.455191    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-data\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.455687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.455670    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-credential-socket\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.456230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.450978    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istiod-ca-cert\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.456416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.455613    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-data\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.456602 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.454178    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-certs\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.456713 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.456002    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-credential-socket\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.456812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.453620    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-socket\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.456936 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.456911    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-podinfo\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.513239 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.513136    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"]
Oct 30 19:12:45.514879 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.514852    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-token\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.515339 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.515309    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tbds\" (UniqueName: \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-kube-api-access-6tbds\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.543596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.541067    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-envoy\") pod \"e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") " pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:45.824464 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:45.824374    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:46.282680 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:46.282635    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:12:46.282794 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:46.282770    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:12:46.282857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:46.282817    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:12:46.282971 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:46.282948    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"]
Oct 30 19:12:47.243255 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:47.243223    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" event={"ID":"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5","Type":"ContainerStarted","Data":"5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d"}
Oct 30 19:12:47.243683 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:47.243274    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" event={"ID":"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5","Type":"ContainerStarted","Data":"5c4a0d8f8bf6699f53e71b3752653c93abb132895c00a2fd42bb8955cc2fe9ad"}
Oct 30 19:12:47.411310 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:47.411249    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" podStartSLOduration=2.411230276 podStartE2EDuration="2.411230276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:47.408720106 +0000 UTC m=+2058.296637197" watchObservedRunningTime="2026-10-30 19:12:47.411230276 +0000 UTC m=+2058.299147367"
Oct 30 19:12:47.828866 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:47.828836    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:47.902535 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:47.902049    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:48.251663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:48.251631    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:48.252504 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:48.252472    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:12:50.161179 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:12:50.161139    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:12:54.026634 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.026590    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"]
Oct 30 19:12:54.038031 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.038005    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:12:54.078274 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.078226    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4tr\" (UniqueName: \"kubernetes.io/projected/8a7d6069-90d8-484b-b9ad-c98dec33c732-kube-api-access-kb4tr\") pod \"payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv\" (UID: \"8a7d6069-90d8-484b-b9ad-c98dec33c732\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:12:54.081762 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.081732    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7d6069-90d8-484b-b9ad-c98dec33c732-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv\" (UID: \"8a7d6069-90d8-484b-b9ad-c98dec33c732\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:12:54.084746 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.084710    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"]
Oct 30 19:12:54.096054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.096029    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8da6e2\""
Oct 30 19:12:54.146683 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.146639    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8da6e2-dockercfg-fsjf9\""
Oct 30 19:12:54.183205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.183161    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4tr\" (UniqueName: \"kubernetes.io/projected/8a7d6069-90d8-484b-b9ad-c98dec33c732-kube-api-access-kb4tr\") pod \"payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv\" (UID: \"8a7d6069-90d8-484b-b9ad-c98dec33c732\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:12:54.183406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.183267    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7d6069-90d8-484b-b9ad-c98dec33c732-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv\" (UID: \"8a7d6069-90d8-484b-b9ad-c98dec33c732\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:12:54.184285 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.184256    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7d6069-90d8-484b-b9ad-c98dec33c732-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv\" (UID: \"8a7d6069-90d8-484b-b9ad-c98dec33c732\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:12:54.239943 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.239909    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4tr\" (UniqueName: \"kubernetes.io/projected/8a7d6069-90d8-484b-b9ad-c98dec33c732-kube-api-access-kb4tr\") pod \"payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv\" (UID: \"8a7d6069-90d8-484b-b9ad-c98dec33c732\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:12:54.397681 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.397599    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:12:54.777015 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:12:54.776983    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7d6069_90d8_484b_b9ad_c98dec33c732.slice/crio-cf6c14ccf5e446d99edc6f19d71fd779d4722954eafbbfeb041cc8f77e9e7a8a WatchSource:0}: Error finding container cf6c14ccf5e446d99edc6f19d71fd779d4722954eafbbfeb041cc8f77e9e7a8a: Status 404 returned error can't find the container with id cf6c14ccf5e446d99edc6f19d71fd779d4722954eafbbfeb041cc8f77e9e7a8a
Oct 30 19:12:54.780413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:54.780387    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"]
Oct 30 19:12:55.310660 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:55.310615    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv" event={"ID":"8a7d6069-90d8-484b-b9ad-c98dec33c732","Type":"ContainerStarted","Data":"c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3"}
Oct 30 19:12:55.311109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:55.310678    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv" event={"ID":"8a7d6069-90d8-484b-b9ad-c98dec33c732","Type":"ContainerStarted","Data":"cf6c14ccf5e446d99edc6f19d71fd779d4722954eafbbfeb041cc8f77e9e7a8a"}
Oct 30 19:12:55.311109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:55.310780    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:12:55.454512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:12:55.448766    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv" podStartSLOduration=2.448742394 podStartE2EDuration="2.448742394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:55.425392176 +0000 UTC m=+2066.313309269" watchObservedRunningTime="2026-10-30 19:12:55.448742394 +0000 UTC m=+2066.336659489"
Oct 30 19:13:06.323516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:06.323468    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:13:10.066018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:10.065979    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"]
Oct 30 19:13:10.066479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:10.066236    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy" containerID="cri-o://7faad4ec3fbd65b5688df1a928cd7f13b53fa1407ebd586a390269530005801a" gracePeriod=30
Oct 30 19:13:10.445891 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:10.445842    4267 patch_prober.go:29] interesting pod/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:10.446099 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:10.445907    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:10.872652 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:10.872566    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"]
Oct 30 19:13:10.872830 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:10.872786    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy" containerID="cri-o://f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e" gracePeriod=30
Oct 30 19:13:11.203112 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:11.203072    4267 patch_prober.go:29] interesting pod/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:11.203548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:11.203129    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:15.547668 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:15.547629    4267 generic.go:356] "Generic (PLEG): container finished" podID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerID="7faad4ec3fbd65b5688df1a928cd7f13b53fa1407ebd586a390269530005801a" exitCode=0
Oct 30 19:13:15.548177 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:15.547703    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" event={"ID":"92e4a69c-3d67-4022-af5b-e768f43f1d85","Type":"ContainerDied","Data":"7faad4ec3fbd65b5688df1a928cd7f13b53fa1407ebd586a390269530005801a"}
Oct 30 19:13:15.804796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:15.804703    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"]
Oct 30 19:13:15.929107 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:15.928771    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b8d0f-htxmd"]
Oct 30 19:13:15.976606 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:13:15.976569    4267 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200f5e70_1d1e_4f6e_9662_a7afd4e3dd7e.slice/crio-f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200f5e70_1d1e_4f6e_9662_a7afd4e3dd7e.slice/crio-conmon-f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:13:16.014275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.014246    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 19:13:16.195542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.195482    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-data\") pod \"92e4a69c-3d67-4022-af5b-e768f43f1d85\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") "
Oct 30 19:13:16.195737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.195633    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-credential-socket\") pod \"92e4a69c-3d67-4022-af5b-e768f43f1d85\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") "
Oct 30 19:13:16.195737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.195697    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e4a69c-3d67-4022-af5b-e768f43f1d85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e4a69c-3d67-4022-af5b-e768f43f1d85-istiod-ca-cert\") pod \"92e4a69c-3d67-4022-af5b-e768f43f1d85\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") "
Oct 30 19:13:16.195865 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.195744    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-certs\") pod \"92e4a69c-3d67-4022-af5b-e768f43f1d85\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") "
Oct 30 19:13:16.195865 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.195770    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-data" pod "92e4a69c-3d67-4022-af5b-e768f43f1d85" (UID: "92e4a69c-3d67-4022-af5b-e768f43f1d85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:16.195865 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.195808    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-podinfo\") pod \"92e4a69c-3d67-4022-af5b-e768f43f1d85\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") "
Oct 30 19:13:16.196018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.195872    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-envoy\") pod \"92e4a69c-3d67-4022-af5b-e768f43f1d85\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") "
Oct 30 19:13:16.196018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.195901    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-credential-socket" pod "92e4a69c-3d67-4022-af5b-e768f43f1d85" (UID: "92e4a69c-3d67-4022-af5b-e768f43f1d85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:16.196018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.195965    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-kube-api-access-x2gsb\" (UniqueName: \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-kube-api-access-x2gsb\") pod \"92e4a69c-3d67-4022-af5b-e768f43f1d85\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") "
Oct 30 19:13:16.196167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.196057    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-token\" (UniqueName: \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-token\") pod \"92e4a69c-3d67-4022-af5b-e768f43f1d85\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") "
Oct 30 19:13:16.196167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.196132    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-certs" pod "92e4a69c-3d67-4022-af5b-e768f43f1d85" (UID: "92e4a69c-3d67-4022-af5b-e768f43f1d85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:16.196167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.196159    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e4a69c-3d67-4022-af5b-e768f43f1d85-istiod-ca-cert" pod "92e4a69c-3d67-4022-af5b-e768f43f1d85" (UID: "92e4a69c-3d67-4022-af5b-e768f43f1d85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:16.196309 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.196226    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-socket\") pod \"92e4a69c-3d67-4022-af5b-e768f43f1d85\" (UID: \"92e4a69c-3d67-4022-af5b-e768f43f1d85\") "
Oct 30 19:13:16.196793 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.196764    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-socket" pod "92e4a69c-3d67-4022-af5b-e768f43f1d85" (UID: "92e4a69c-3d67-4022-af5b-e768f43f1d85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:16.197062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.197035    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.197231 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.197064    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.197231 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.197082    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.197231 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.197096    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92e4a69c-3d67-4022-af5b-e768f43f1d85-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.197231 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.197110    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.204679 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.198802    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-podinfo" pod "92e4a69c-3d67-4022-af5b-e768f43f1d85" (UID: "92e4a69c-3d67-4022-af5b-e768f43f1d85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:16.204679 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.199146    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-envoy" pod "92e4a69c-3d67-4022-af5b-e768f43f1d85" (UID: "92e4a69c-3d67-4022-af5b-e768f43f1d85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:16.204679 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.199227    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-kube-api-access-x2gsb" pod "92e4a69c-3d67-4022-af5b-e768f43f1d85" (UID: "92e4a69c-3d67-4022-af5b-e768f43f1d85"). InnerVolumeSpecName "kube-api-access-x2gsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:16.204679 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.199351    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-token" pod "92e4a69c-3d67-4022-af5b-e768f43f1d85" (UID: "92e4a69c-3d67-4022-af5b-e768f43f1d85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:16.225126 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.225098    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 19:13:16.298727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.298696    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.298727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.298724    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gsb\" (UniqueName: \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-kube-api-access-x2gsb\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.298727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.298736    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.298992 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.298745    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92e4a69c-3d67-4022-af5b-e768f43f1d85-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.400098 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400002    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-envoy\") pod \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") "
Oct 30 19:13:16.400098 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400061    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-podinfo\") pod \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") "
Oct 30 19:13:16.400098 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400094    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-kube-api-access-n4xcz\" (UniqueName: \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-kube-api-access-n4xcz\") pod \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") "
Oct 30 19:13:16.400386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400194    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-certs\") pod \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") "
Oct 30 19:13:16.400386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400254    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-credential-socket\") pod \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") "
Oct 30 19:13:16.400386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400326    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-socket\") pod \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") "
Oct 30 19:13:16.400386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400357    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istiod-ca-cert\") pod \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") "
Oct 30 19:13:16.400620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400460    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-data\") pod \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") "
Oct 30 19:13:16.400620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400565    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-credential-socket" pod "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" (UID: "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:16.400620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400606    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-token\" (UniqueName: \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-token\") pod \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\" (UID: \"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e\") "
Oct 30 19:13:16.400778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400726    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-socket" pod "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" (UID: "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:16.400778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.400762    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-certs" pod "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" (UID: "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:16.401246 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.401224    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-data" pod "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" (UID: "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:16.401521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.401483    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.401594 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.401528    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.401594 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.401545    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.401594 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.401559    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.403959 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.403350    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istiod-ca-cert" pod "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" (UID: "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:16.403959 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.403487    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-kube-api-access-n4xcz" pod "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" (UID: "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e"). InnerVolumeSpecName "kube-api-access-n4xcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:16.404610 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.404586    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-podinfo" pod "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" (UID: "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:16.410563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.405299    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-token" pod "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" (UID: "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:16.413641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.413600    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-envoy" pod "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" (UID: "200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:16.503046 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.503001    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.503046 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.503048    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.503312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.503062    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.503312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.503077    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.503312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.503091    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4xcz\" (UniqueName: \"kubernetes.io/projected/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e-kube-api-access-n4xcz\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:16.565321 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.565262    4267 generic.go:356] "Generic (PLEG): container finished" podID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerID="f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e" exitCode=0
Oct 30 19:13:16.565874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.565398    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" event={"ID":"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e","Type":"ContainerDied","Data":"f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e"}
Oct 30 19:13:16.565874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.565482    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl" event={"ID":"200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e","Type":"ContainerDied","Data":"23c482d350730631e5e6d6f304252bc5330f56cb0b5c3073809bbd38736764d9"}
Oct 30 19:13:16.565874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.565532    4267 scope.go:122] "RemoveContainer" containerID="f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e"
Oct 30 19:13:16.565874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.565701    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"
Oct 30 19:13:16.575139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.575110    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl" event={"ID":"92e4a69c-3d67-4022-af5b-e768f43f1d85","Type":"ContainerDied","Data":"3f70ea2aae1f106dae6faa581a50e79850f02f0e8e7bf52481c1a6da79205b0e"}
Oct 30 19:13:16.575304 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.575219    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"
Oct 30 19:13:16.583261 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.583236    4267 scope.go:122] "RemoveContainer" containerID="f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e"
Oct 30 19:13:16.583638 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:13:16.583615    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e\": container with ID starting with f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e not found: ID does not exist" containerID="f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e"
Oct 30 19:13:16.583743 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.583647    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e"} err="failed to get container status \"f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e\": rpc error: code = NotFound desc = could not find container \"f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e\": container with ID starting with f44d7e73c9532d3ba5f40d9464fe2d0038d936a3fa3d713552b25036f4bced6e not found: ID does not exist"
Oct 30 19:13:16.583743 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.583665    4267 scope.go:122] "RemoveContainer" containerID="7faad4ec3fbd65b5688df1a928cd7f13b53fa1407ebd586a390269530005801a"
Oct 30 19:13:16.749370 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.749328    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"]
Oct 30 19:13:16.787832 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.787792    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-xb9vl"]
Oct 30 19:13:16.907549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.906395    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"]
Oct 30 19:13:16.918524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:16.915182    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-7tkxl"]
Oct 30 19:13:17.101807 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:17.101721    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"]
Oct 30 19:13:17.106177 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:17.106151    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-q29zw"]
Oct 30 19:13:17.225440 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:17.225409    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" path="/var/lib/kubelet/pods/200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e/volumes"
Oct 30 19:13:17.225995 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:17.225971    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" path="/var/lib/kubelet/pods/92e4a69c-3d67-4022-af5b-e768f43f1d85/volumes"
Oct 30 19:13:17.226522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:17.226486    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e746d4ba-25cf-4492-a900-71f8702ae6e9" path="/var/lib/kubelet/pods/e746d4ba-25cf-4492-a900-71f8702ae6e9/volumes"
Oct 30 19:13:17.227335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:17.227312    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8875ee2-dc16-45e7-a136-72bb59a97eb0" path="/var/lib/kubelet/pods/e8875ee2-dc16-45e7-a136-72bb59a97eb0/volumes"
Oct 30 19:13:20.159100 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:13:20.159060    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:13:29.839549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.839510    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"]
Oct 30 19:13:29.840222 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.840206    4267 cpu_manager.go:414] "containerMap: removing container" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy"
Oct 30 19:13:29.840222 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.840221    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy"
Oct 30 19:13:29.840341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.840261    4267 cpu_manager.go:414] "containerMap: removing container" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy"
Oct 30 19:13:29.840341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.840268    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy"
Oct 30 19:13:29.840450 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.840360    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e4a69c-3d67-4022-af5b-e768f43f1d85" containerName="istio-proxy"
Oct 30 19:13:29.840450 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.840373    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="200f5e70-1d1e-4f6e-9662-a7afd4e3dd7e" containerName="istio-proxy"
Oct 30 19:13:29.842932 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.842910    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:29.900477 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.900439    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"]
Oct 30 19:13:29.945278 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.945130    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:29.945278 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.945240    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:13:29.962872 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.962840    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-322e97-4r44l\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:29.963047 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.963025    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn25q\" (UniqueName: \"kubernetes.io/projected/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-kube-api-access-gn25q\") pod \"maas-api-revoke-keys-e2e-inf-a-322e97-4r44l\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:29.963149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:29.963134    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-322e97-4r44l\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:30.064906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:30.064872    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn25q\" (UniqueName: \"kubernetes.io/projected/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-kube-api-access-gn25q\") pod \"maas-api-revoke-keys-e2e-inf-a-322e97-4r44l\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:30.065067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:30.064963    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-322e97-4r44l\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:30.065150 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:30.065067    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-322e97-4r44l\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:30.065724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:30.065705    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-322e97-4r44l\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:30.067311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:30.067285    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-322e97-4r44l\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:30.121974 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:30.121891    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn25q\" (UniqueName: \"kubernetes.io/projected/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-kube-api-access-gn25q\") pod \"maas-api-revoke-keys-e2e-inf-a-322e97-4r44l\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:30.163996 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:30.163969    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:30.587944 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:30.587913    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"]
Oct 30 19:13:30.596699 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:13:30.596656    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf4ac3e_7f60_4006_8346_f95ad7a51a48.slice/crio-d16c689661119ba5b1f1a1f06d3b67008aa472a21c649ab70f880cdb269b8e3c WatchSource:0}: Error finding container d16c689661119ba5b1f1a1f06d3b67008aa472a21c649ab70f880cdb269b8e3c: Status 404 returned error can't find the container with id d16c689661119ba5b1f1a1f06d3b67008aa472a21c649ab70f880cdb269b8e3c
Oct 30 19:13:30.788362 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:30.788321    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l" event={"ID":"fbf4ac3e-7f60-4006-8346-f95ad7a51a48","Type":"ContainerStarted","Data":"d16c689661119ba5b1f1a1f06d3b67008aa472a21c649ab70f880cdb269b8e3c"}
Oct 30 19:13:31.798689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:31.798656    4267 generic.go:356] "Generic (PLEG): container finished" podID="fbf4ac3e-7f60-4006-8346-f95ad7a51a48" containerID="42a3e5c7393fa18d53f15c5419f0aae9d32365581af5c5038889e3ac15e47697" exitCode=0
Oct 30 19:13:31.799052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:31.798739    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l" event={"ID":"fbf4ac3e-7f60-4006-8346-f95ad7a51a48","Type":"ContainerDied","Data":"42a3e5c7393fa18d53f15c5419f0aae9d32365581af5c5038889e3ac15e47697"}
Oct 30 19:13:32.980874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:32.980842    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:33.124165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.124077    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-openshift-ca\") pod \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") "
Oct 30 19:13:33.124165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.124162    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-kube-api-access-gn25q\" (UniqueName: \"kubernetes.io/projected/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-kube-api-access-gn25q\") pod \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") "
Oct 30 19:13:33.124404 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.124371    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-xks-ca\") pod \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\" (UID: \"fbf4ac3e-7f60-4006-8346-f95ad7a51a48\") "
Oct 30 19:13:33.124521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.124471    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-openshift-ca" pod "fbf4ac3e-7f60-4006-8346-f95ad7a51a48" (UID: "fbf4ac3e-7f60-4006-8346-f95ad7a51a48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:33.125152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.125124    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:33.132696 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.132450    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-kube-api-access-gn25q" pod "fbf4ac3e-7f60-4006-8346-f95ad7a51a48" (UID: "fbf4ac3e-7f60-4006-8346-f95ad7a51a48"). InnerVolumeSpecName "kube-api-access-gn25q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:33.133699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.133670    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-xks-ca" pod "fbf4ac3e-7f60-4006-8346-f95ad7a51a48" (UID: "fbf4ac3e-7f60-4006-8346-f95ad7a51a48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:33.226248 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.226217    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn25q\" (UniqueName: \"kubernetes.io/projected/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-kube-api-access-gn25q\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:33.226248 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.226245    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf4ac3e-7f60-4006-8346-f95ad7a51a48-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:33.819237 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.819203    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"
Oct 30 19:13:33.819429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.819207    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l" event={"ID":"fbf4ac3e-7f60-4006-8346-f95ad7a51a48","Type":"ContainerDied","Data":"d16c689661119ba5b1f1a1f06d3b67008aa472a21c649ab70f880cdb269b8e3c"}
Oct 30 19:13:33.819429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:33.819334    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d16c689661119ba5b1f1a1f06d3b67008aa472a21c649ab70f880cdb269b8e3c"
Oct 30 19:13:36.818370 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.818329    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"]
Oct 30 19:13:36.819458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.819429    4267 cpu_manager.go:414] "containerMap: removing container" podUID="fbf4ac3e-7f60-4006-8346-f95ad7a51a48" containerName="revoke-keys"
Oct 30 19:13:36.819526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.819461    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf4ac3e-7f60-4006-8346-f95ad7a51a48" containerName="revoke-keys"
Oct 30 19:13:36.819702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.819689    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf4ac3e-7f60-4006-8346-f95ad7a51a48" containerName="revoke-keys"
Oct 30 19:13:36.823051 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.823032    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:36.824136 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.824102    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:36.824361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.824335    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:36.824539 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.824479    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqht\" (UniqueName: \"kubernetes.io/projected/c79b312b-9ae3-4371-9a77-8cdfc8583c47-kube-api-access-bgqht\") pod \"maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:36.875312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.875251    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:36.930905 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.930857    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:36.933786 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.933758    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:36.934561 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.934535    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqht\" (UniqueName: \"kubernetes.io/projected/c79b312b-9ae3-4371-9a77-8cdfc8583c47-kube-api-access-bgqht\") pod \"maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:36.951779 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.931298    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:13:36.952015 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.933542    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:36.952163 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.935473    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"]
Oct 30 19:13:36.952300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.946976    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:36.992244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:36.992212    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqht\" (UniqueName: \"kubernetes.io/projected/c79b312b-9ae3-4371-9a77-8cdfc8583c47-kube-api-access-bgqht\") pod \"maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:37.139119 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:37.139028    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:37.523102 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:13:37.523066    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79b312b_9ae3_4371_9a77_8cdfc8583c47.slice/crio-763039d479549ab69720cba0908b1e48f403ee8e60b83102b2c805a8b426bff2 WatchSource:0}: Error finding container 763039d479549ab69720cba0908b1e48f403ee8e60b83102b2c805a8b426bff2: Status 404 returned error can't find the container with id 763039d479549ab69720cba0908b1e48f403ee8e60b83102b2c805a8b426bff2
Oct 30 19:13:37.523687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:37.523660    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"]
Oct 30 19:13:37.993151 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:37.993096    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8" event={"ID":"c79b312b-9ae3-4371-9a77-8cdfc8583c47","Type":"ContainerStarted","Data":"9594f6677158ded42b76e72947aac8d9e7274f56871e4db5ff893c626857514c"}
Oct 30 19:13:37.993675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:37.993171    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8" event={"ID":"c79b312b-9ae3-4371-9a77-8cdfc8583c47","Type":"ContainerStarted","Data":"763039d479549ab69720cba0908b1e48f403ee8e60b83102b2c805a8b426bff2"}
Oct 30 19:13:38.156397 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:38.156277    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8" podStartSLOduration=2.156262596 podStartE2EDuration="2.156262596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:38.105627599 +0000 UTC m=+2108.993544692" watchObservedRunningTime="2026-10-30 19:13:38.156262596 +0000 UTC m=+2109.044179684"
Oct 30 19:13:39.002089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:39.002051    4267 generic.go:356] "Generic (PLEG): container finished" podID="c79b312b-9ae3-4371-9a77-8cdfc8583c47" containerID="9594f6677158ded42b76e72947aac8d9e7274f56871e4db5ff893c626857514c" exitCode=0
Oct 30 19:13:39.002552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:39.002131    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8" event={"ID":"c79b312b-9ae3-4371-9a77-8cdfc8583c47","Type":"ContainerDied","Data":"9594f6677158ded42b76e72947aac8d9e7274f56871e4db5ff893c626857514c"}
Oct 30 19:13:40.191071 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:40.190667    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:40.341573 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:40.341464    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79b312b-9ae3-4371-9a77-8cdfc8583c47-kube-api-access-bgqht\" (UniqueName: \"kubernetes.io/projected/c79b312b-9ae3-4371-9a77-8cdfc8583c47-kube-api-access-bgqht\") pod \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") "
Oct 30 19:13:40.341795 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:40.341661    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-openshift-ca\") pod \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") "
Oct 30 19:13:40.341795 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:40.341715    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-xks-ca\") pod \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\" (UID: \"c79b312b-9ae3-4371-9a77-8cdfc8583c47\") "
Oct 30 19:13:40.376886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:40.376834    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-xks-ca" pod "c79b312b-9ae3-4371-9a77-8cdfc8583c47" (UID: "c79b312b-9ae3-4371-9a77-8cdfc8583c47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:40.394841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:40.394797    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-openshift-ca" pod "c79b312b-9ae3-4371-9a77-8cdfc8583c47" (UID: "c79b312b-9ae3-4371-9a77-8cdfc8583c47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:40.401371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:40.401150    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79b312b-9ae3-4371-9a77-8cdfc8583c47-kube-api-access-bgqht" pod "c79b312b-9ae3-4371-9a77-8cdfc8583c47" (UID: "c79b312b-9ae3-4371-9a77-8cdfc8583c47"). InnerVolumeSpecName "kube-api-access-bgqht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:40.445042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:40.444988    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqht\" (UniqueName: \"kubernetes.io/projected/c79b312b-9ae3-4371-9a77-8cdfc8583c47-kube-api-access-bgqht\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:40.445042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:40.445035    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:40.445309 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:40.445053    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79b312b-9ae3-4371-9a77-8cdfc8583c47-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:41.022094 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:41.022011    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"
Oct 30 19:13:41.022094 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:41.022032    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8" event={"ID":"c79b312b-9ae3-4371-9a77-8cdfc8583c47","Type":"ContainerDied","Data":"763039d479549ab69720cba0908b1e48f403ee8e60b83102b2c805a8b426bff2"}
Oct 30 19:13:41.022094 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:41.022094    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="763039d479549ab69720cba0908b1e48f403ee8e60b83102b2c805a8b426bff2"
Oct 30 19:13:45.756294 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:45.756239    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"]
Oct 30 19:13:45.874735 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:45.874698    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"]
Oct 30 19:13:50.230185 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:13:50.230144    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:13:50.756845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:50.756787    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk" podUID="90234b3d-9ec5-4d5d-b1ee-8b517d3c5229" containerName="payload-processing" containerID="cri-o://bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0" gracePeriod=25
Oct 30 19:13:50.875473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:50.875412    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s" podUID="8b31d077-8101-44f4-86a6-c03845acda3e" containerName="payload-pre-processing" containerID="cri-o://aef33eab280b5904ed2ed25246d5513fa7bbf4af9dc4eb99d6d7750f359fc4d8" gracePeriod=25
Oct 30 19:13:51.076479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.076451    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:13:51.184664 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.184622    4267 generic.go:356] "Generic (PLEG): container finished" podID="8b31d077-8101-44f4-86a6-c03845acda3e" containerID="aef33eab280b5904ed2ed25246d5513fa7bbf4af9dc4eb99d6d7750f359fc4d8" exitCode=0
Oct 30 19:13:51.184829 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.184790    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s" event={"ID":"8b31d077-8101-44f4-86a6-c03845acda3e","Type":"ContainerDied","Data":"aef33eab280b5904ed2ed25246d5513fa7bbf4af9dc4eb99d6d7750f359fc4d8"}
Oct 30 19:13:51.184897 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.184881    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s" event={"ID":"8b31d077-8101-44f4-86a6-c03845acda3e","Type":"ContainerDied","Data":"f1a0952e44c86bc054155a2170c65dbc8f3bc63f07267aaeb9eedbade3676696"}
Oct 30 19:13:51.184937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.184912    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1a0952e44c86bc054155a2170c65dbc8f3bc63f07267aaeb9eedbade3676696"
Oct 30 19:13:51.186374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.186331    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:13:51.187159 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.187132    4267 generic.go:356] "Generic (PLEG): container finished" podID="90234b3d-9ec5-4d5d-b1ee-8b517d3c5229" containerID="bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0" exitCode=0
Oct 30 19:13:51.187296 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.187191    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"
Oct 30 19:13:51.187296 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.187245    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk" event={"ID":"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229","Type":"ContainerDied","Data":"bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0"}
Oct 30 19:13:51.187296 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.187287    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk" event={"ID":"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229","Type":"ContainerDied","Data":"210572982a06d7b0445965f8c48ec03d25893c568a355cf08d33bb74f9a64340"}
Oct 30 19:13:51.187410 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.187315    4267 scope.go:122] "RemoveContainer" containerID="bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0"
Oct 30 19:13:51.200505 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.200470    4267 scope.go:122] "RemoveContainer" containerID="bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0"
Oct 30 19:13:51.200800 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:13:51.200780    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0\": container with ID starting with bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0 not found: ID does not exist" containerID="bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0"
Oct 30 19:13:51.200889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.200808    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0"} err="failed to get container status \"bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0\": rpc error: code = NotFound desc = could not find container \"bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0\": container with ID starting with bd86df0df8522e8a9bf4beaa890fa42eb45cea65478a8e826e9bc1f3a79db4b0 not found: ID does not exist"
Oct 30 19:13:51.225207 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.225171    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-plugins-config-volume\") pod \"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229\" (UID: \"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229\") "
Oct 30 19:13:51.225396 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.225373    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-kube-api-access-ph5r5\" (UniqueName: \"kubernetes.io/projected/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-kube-api-access-ph5r5\") pod \"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229\" (UID: \"90234b3d-9ec5-4d5d-b1ee-8b517d3c5229\") "
Oct 30 19:13:51.225659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.225627    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-plugins-config-volume" pod "90234b3d-9ec5-4d5d-b1ee-8b517d3c5229" (UID: "90234b3d-9ec5-4d5d-b1ee-8b517d3c5229"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:51.226422 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.226390    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:51.228813 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.228780    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-kube-api-access-ph5r5" pod "90234b3d-9ec5-4d5d-b1ee-8b517d3c5229" (UID: "90234b3d-9ec5-4d5d-b1ee-8b517d3c5229"). InnerVolumeSpecName "kube-api-access-ph5r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:51.327431 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.327335    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b31d077-8101-44f4-86a6-c03845acda3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b31d077-8101-44f4-86a6-c03845acda3e-plugins-config-volume\") pod \"8b31d077-8101-44f4-86a6-c03845acda3e\" (UID: \"8b31d077-8101-44f4-86a6-c03845acda3e\") "
Oct 30 19:13:51.327431 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.327411    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b31d077-8101-44f4-86a6-c03845acda3e-kube-api-access-crhd6\" (UniqueName: \"kubernetes.io/projected/8b31d077-8101-44f4-86a6-c03845acda3e-kube-api-access-crhd6\") pod \"8b31d077-8101-44f4-86a6-c03845acda3e\" (UID: \"8b31d077-8101-44f4-86a6-c03845acda3e\") "
Oct 30 19:13:51.328194 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.328172    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph5r5\" (UniqueName: \"kubernetes.io/projected/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229-kube-api-access-ph5r5\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:51.329240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.328754    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b31d077-8101-44f4-86a6-c03845acda3e-plugins-config-volume" pod "8b31d077-8101-44f4-86a6-c03845acda3e" (UID: "8b31d077-8101-44f4-86a6-c03845acda3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:51.331204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.331168    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b31d077-8101-44f4-86a6-c03845acda3e-kube-api-access-crhd6" pod "8b31d077-8101-44f4-86a6-c03845acda3e" (UID: "8b31d077-8101-44f4-86a6-c03845acda3e"). InnerVolumeSpecName "kube-api-access-crhd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:51.429259 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.429223    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b31d077-8101-44f4-86a6-c03845acda3e-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:51.429259 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.429253    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crhd6\" (UniqueName: \"kubernetes.io/projected/8b31d077-8101-44f4-86a6-c03845acda3e-kube-api-access-crhd6\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:13:51.604021 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.603914    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"]
Oct 30 19:13:51.609262 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:51.609227    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-322e97-dfc4bc46f-7gwhk"]
Oct 30 19:13:52.198067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:52.198036    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"
Oct 30 19:13:52.363913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:52.363871    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"]
Oct 30 19:13:52.368240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:52.368213    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-322e97-7bf994878c-d8q7s"]
Oct 30 19:13:53.225981 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:53.225945    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b31d077-8101-44f4-86a6-c03845acda3e" path="/var/lib/kubelet/pods/8b31d077-8101-44f4-86a6-c03845acda3e/volumes"
Oct 30 19:13:53.226616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:13:53.226596    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90234b3d-9ec5-4d5d-b1ee-8b517d3c5229" path="/var/lib/kubelet/pods/90234b3d-9ec5-4d5d-b1ee-8b517d3c5229/volumes"
Oct 30 19:14:04.074519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:04.073639    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"]
Oct 30 19:14:04.105279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:04.105242    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e261a7fd-dthgb"]
Oct 30 19:14:05.225988 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:05.225941    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d25088-7507-4245-b9ca-7b9f18a03dc2" path="/var/lib/kubelet/pods/22d25088-7507-4245-b9ca-7b9f18a03dc2/volumes"
Oct 30 19:14:05.797076 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:05.797034    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"]
Oct 30 19:14:05.843850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:05.843814    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e004a-2vf9r"]
Oct 30 19:14:07.223479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:07.223447    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="958a790f-dd68-46f5-ae31-7f31a976ba2d" path="/var/lib/kubelet/pods/958a790f-dd68-46f5-ae31-7f31a976ba2d/volumes"
Oct 30 19:14:10.887930 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:10.887892    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"]
Oct 30 19:14:10.888350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:10.888092    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy" containerID="cri-o://af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d" gracePeriod=30
Oct 30 19:14:14.425073 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.425039    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"]
Oct 30 19:14:14.425736 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.425721    4267 cpu_manager.go:414] "containerMap: removing container" podUID="90234b3d-9ec5-4d5d-b1ee-8b517d3c5229" containerName="payload-processing"
Oct 30 19:14:14.425790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.425736    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90234b3d-9ec5-4d5d-b1ee-8b517d3c5229" containerName="payload-processing"
Oct 30 19:14:14.425790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.425758    4267 cpu_manager.go:414] "containerMap: removing container" podUID="8b31d077-8101-44f4-86a6-c03845acda3e" containerName="payload-pre-processing"
Oct 30 19:14:14.425790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.425763    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b31d077-8101-44f4-86a6-c03845acda3e" containerName="payload-pre-processing"
Oct 30 19:14:14.425790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.425778    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c79b312b-9ae3-4371-9a77-8cdfc8583c47" containerName="revoke-keys"
Oct 30 19:14:14.425790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.425783    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c79b312b-9ae3-4371-9a77-8cdfc8583c47" containerName="revoke-keys"
Oct 30 19:14:14.425950 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.425895    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b31d077-8101-44f4-86a6-c03845acda3e" containerName="payload-pre-processing"
Oct 30 19:14:14.425950 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.425906    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c79b312b-9ae3-4371-9a77-8cdfc8583c47" containerName="revoke-keys"
Oct 30 19:14:14.425950 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.425918    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="90234b3d-9ec5-4d5d-b1ee-8b517d3c5229" containerName="payload-processing"
Oct 30 19:14:14.430726 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.430704    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:14.485261 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.485229    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"]
Oct 30 19:14:14.529575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.529542    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:14.529806 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.529578    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:14:14.607215 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.607181    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:14.607422 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.607400    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:14.607480 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.607452    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgcpg\" (UniqueName: \"kubernetes.io/projected/defecc52-428f-4f15-acef-aabfb38aa43f-kube-api-access-bgcpg\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:14.709103 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.709063    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:14.709308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.709249    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:14.709308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.709279    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcpg\" (UniqueName: \"kubernetes.io/projected/defecc52-428f-4f15-acef-aabfb38aa43f-kube-api-access-bgcpg\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:14.709980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.709955    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:14.712105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.712075    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:14.766513 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:14.766465    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgcpg\" (UniqueName: \"kubernetes.io/projected/defecc52-428f-4f15-acef-aabfb38aa43f-kube-api-access-bgcpg\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:15.047791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:15.047701    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:15.429135 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:14:15.429092    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefecc52_428f_4f15_acef_aabfb38aa43f.slice/crio-b4a09e2e4b3baf0b0778c86b77df4b2b1f4636c5d4cc79b17fa5f1bba01f6487 WatchSource:0}: Error finding container b4a09e2e4b3baf0b0778c86b77df4b2b1f4636c5d4cc79b17fa5f1bba01f6487: Status 404 returned error can't find the container with id b4a09e2e4b3baf0b0778c86b77df4b2b1f4636c5d4cc79b17fa5f1bba01f6487
Oct 30 19:14:15.431098 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:15.431024    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"]
Oct 30 19:14:16.249353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.249327    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:14:16.250777 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.250756    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75951caa-ff7b-4c5b-b55b-11a66a055f17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75951caa-ff7b-4c5b-b55b-11a66a055f17-istiod-ca-cert\") pod \"75951caa-ff7b-4c5b-b55b-11a66a055f17\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") "
Oct 30 19:14:16.250868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.250802    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-credential-socket\") pod \"75951caa-ff7b-4c5b-b55b-11a66a055f17\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") "
Oct 30 19:14:16.250929 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.250873    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-kube-api-access-bdzjc\" (UniqueName: \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-kube-api-access-bdzjc\") pod \"75951caa-ff7b-4c5b-b55b-11a66a055f17\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") "
Oct 30 19:14:16.250985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.250939    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-socket\") pod \"75951caa-ff7b-4c5b-b55b-11a66a055f17\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") "
Oct 30 19:14:16.250985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.250962    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-envoy\") pod \"75951caa-ff7b-4c5b-b55b-11a66a055f17\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") "
Oct 30 19:14:16.251098 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.251037    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-token\" (UniqueName: \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-token\") pod \"75951caa-ff7b-4c5b-b55b-11a66a055f17\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") "
Oct 30 19:14:16.251155 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.251103    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-data\") pod \"75951caa-ff7b-4c5b-b55b-11a66a055f17\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") "
Oct 30 19:14:16.251213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.251169    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-podinfo\") pod \"75951caa-ff7b-4c5b-b55b-11a66a055f17\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") "
Oct 30 19:14:16.251306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.251287    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-certs\") pod \"75951caa-ff7b-4c5b-b55b-11a66a055f17\" (UID: \"75951caa-ff7b-4c5b-b55b-11a66a055f17\") "
Oct 30 19:14:16.251674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.251165    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-credential-socket" pod "75951caa-ff7b-4c5b-b55b-11a66a055f17" (UID: "75951caa-ff7b-4c5b-b55b-11a66a055f17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:16.251674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.251260    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75951caa-ff7b-4c5b-b55b-11a66a055f17-istiod-ca-cert" pod "75951caa-ff7b-4c5b-b55b-11a66a055f17" (UID: "75951caa-ff7b-4c5b-b55b-11a66a055f17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:16.251866 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.251723    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-data" pod "75951caa-ff7b-4c5b-b55b-11a66a055f17" (UID: "75951caa-ff7b-4c5b-b55b-11a66a055f17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:16.251992 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.251969    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-certs" pod "75951caa-ff7b-4c5b-b55b-11a66a055f17" (UID: "75951caa-ff7b-4c5b-b55b-11a66a055f17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:16.252063 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.252032    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-socket" pod "75951caa-ff7b-4c5b-b55b-11a66a055f17" (UID: "75951caa-ff7b-4c5b-b55b-11a66a055f17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:16.252813 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.252789    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.252813 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.252813    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75951caa-ff7b-4c5b-b55b-11a66a055f17-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.252987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.252823    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.252987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.252833    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.252987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.252841    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.253757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.253729    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-envoy" pod "75951caa-ff7b-4c5b-b55b-11a66a055f17" (UID: "75951caa-ff7b-4c5b-b55b-11a66a055f17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:16.254074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.254042    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-kube-api-access-bdzjc" pod "75951caa-ff7b-4c5b-b55b-11a66a055f17" (UID: "75951caa-ff7b-4c5b-b55b-11a66a055f17"). InnerVolumeSpecName "kube-api-access-bdzjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:16.254074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.254060    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-podinfo" pod "75951caa-ff7b-4c5b-b55b-11a66a055f17" (UID: "75951caa-ff7b-4c5b-b55b-11a66a055f17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:16.254458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.254432    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-token" pod "75951caa-ff7b-4c5b-b55b-11a66a055f17" (UID: "75951caa-ff7b-4c5b-b55b-11a66a055f17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:16.355025 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.354986    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.355025 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.355023    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.355233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.355039    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75951caa-ff7b-4c5b-b55b-11a66a055f17-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.355233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.355053    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdzjc\" (UniqueName: \"kubernetes.io/projected/75951caa-ff7b-4c5b-b55b-11a66a055f17-kube-api-access-bdzjc\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.418047 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.418007    4267 generic.go:356] "Generic (PLEG): container finished" podID="defecc52-428f-4f15-acef-aabfb38aa43f" containerID="dbe4b33145a6a3f5133173288e954df038c6d0e6aee70dfe119cca41f6071eb8" exitCode=0
Oct 30 19:14:16.418284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.418096    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7" event={"ID":"defecc52-428f-4f15-acef-aabfb38aa43f","Type":"ContainerDied","Data":"dbe4b33145a6a3f5133173288e954df038c6d0e6aee70dfe119cca41f6071eb8"}
Oct 30 19:14:16.418284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.418197    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7" event={"ID":"defecc52-428f-4f15-acef-aabfb38aa43f","Type":"ContainerStarted","Data":"b4a09e2e4b3baf0b0778c86b77df4b2b1f4636c5d4cc79b17fa5f1bba01f6487"}
Oct 30 19:14:16.420186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.420156    4267 generic.go:356] "Generic (PLEG): container finished" podID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerID="af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d" exitCode=0
Oct 30 19:14:16.420341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.420235    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" event={"ID":"75951caa-ff7b-4c5b-b55b-11a66a055f17","Type":"ContainerDied","Data":"af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d"}
Oct 30 19:14:16.420341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.420274    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm" event={"ID":"75951caa-ff7b-4c5b-b55b-11a66a055f17","Type":"ContainerDied","Data":"f17b78de18ad575c55f9ac3342e9f8b31467f89014f7270a4730d0218740c98e"}
Oct 30 19:14:16.420341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.420301    4267 scope.go:122] "RemoveContainer" containerID="af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d"
Oct 30 19:14:16.420532 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.420455    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"
Oct 30 19:14:16.434757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.434734    4267 scope.go:122] "RemoveContainer" containerID="af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d"
Oct 30 19:14:16.435121 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:14:16.435053    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d\": container with ID starting with af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d not found: ID does not exist" containerID="af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d"
Oct 30 19:14:16.435121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.435086    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d"} err="failed to get container status \"af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d\": rpc error: code = NotFound desc = could not find container \"af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d\": container with ID starting with af9e397f9cfd18e71fba050c5025949de40757c66eb7b2f19f8ea43253f2f08d not found: ID does not exist"
Oct 30 19:14:16.657910 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.657817    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"]
Oct 30 19:14:16.662105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:16.662079    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-rj2fm"]
Oct 30 19:14:17.224239 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.224199    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" path="/var/lib/kubelet/pods/75951caa-ff7b-4c5b-b55b-11a66a055f17/volumes"
Oct 30 19:14:17.606958 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.606931    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:17.674539 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.674473    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-xks-ca\") pod \"defecc52-428f-4f15-acef-aabfb38aa43f\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") "
Oct 30 19:14:17.674698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.674671    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defecc52-428f-4f15-acef-aabfb38aa43f-kube-api-access-bgcpg\" (UniqueName: \"kubernetes.io/projected/defecc52-428f-4f15-acef-aabfb38aa43f-kube-api-access-bgcpg\") pod \"defecc52-428f-4f15-acef-aabfb38aa43f\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") "
Oct 30 19:14:17.674762 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.674712    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-openshift-ca\") pod \"defecc52-428f-4f15-acef-aabfb38aa43f\" (UID: \"defecc52-428f-4f15-acef-aabfb38aa43f\") "
Oct 30 19:14:17.675133 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.675105    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-openshift-ca" pod "defecc52-428f-4f15-acef-aabfb38aa43f" (UID: "defecc52-428f-4f15-acef-aabfb38aa43f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:17.675339 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.675320    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:17.676817 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.676785    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defecc52-428f-4f15-acef-aabfb38aa43f-kube-api-access-bgcpg" pod "defecc52-428f-4f15-acef-aabfb38aa43f" (UID: "defecc52-428f-4f15-acef-aabfb38aa43f"). InnerVolumeSpecName "kube-api-access-bgcpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:17.676915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.676816    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-xks-ca" pod "defecc52-428f-4f15-acef-aabfb38aa43f" (UID: "defecc52-428f-4f15-acef-aabfb38aa43f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:17.776810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.776710    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defecc52-428f-4f15-acef-aabfb38aa43f-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:17.776810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:17.776742    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgcpg\" (UniqueName: \"kubernetes.io/projected/defecc52-428f-4f15-acef-aabfb38aa43f-kube-api-access-bgcpg\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:18.443099 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:18.443066    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"
Oct 30 19:14:18.443099 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:18.443086    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7" event={"ID":"defecc52-428f-4f15-acef-aabfb38aa43f","Type":"ContainerDied","Data":"b4a09e2e4b3baf0b0778c86b77df4b2b1f4636c5d4cc79b17fa5f1bba01f6487"}
Oct 30 19:14:18.443317 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:18.443141    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4a09e2e4b3baf0b0778c86b77df4b2b1f4636c5d4cc79b17fa5f1bba01f6487"
Oct 30 19:14:19.954923 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:19.954884    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"]
Oct 30 19:14:19.955442 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:19.955142    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" podUID="fea86300-bcfd-4c31-893d-342130676629" containerName="istio-proxy" containerID="cri-o://88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc" gracePeriod=30
Oct 30 19:14:20.171473 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:14:20.171427    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:14:25.240841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.240812    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:14:25.293926 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.293891    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-certs\") pod \"fea86300-bcfd-4c31-893d-342130676629\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") "
Oct 30 19:14:25.294124 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.293936    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fea86300-bcfd-4c31-893d-342130676629-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea86300-bcfd-4c31-893d-342130676629-istio-podinfo\") pod \"fea86300-bcfd-4c31-893d-342130676629\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") "
Oct 30 19:14:25.294124 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294029    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-istio-token\" (UniqueName: \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-istio-token\") pod \"fea86300-bcfd-4c31-893d-342130676629\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") "
Oct 30 19:14:25.294235 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294167    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-credential-socket\") pod \"fea86300-bcfd-4c31-893d-342130676629\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") "
Oct 30 19:14:25.294235 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294183    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-certs" pod "fea86300-bcfd-4c31-893d-342130676629" (UID: "fea86300-bcfd-4c31-893d-342130676629"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:25.294235 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294207    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-data\") pod \"fea86300-bcfd-4c31-893d-342130676629\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") "
Oct 30 19:14:25.294371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294259    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-kube-api-access-gzcrw\" (UniqueName: \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-kube-api-access-gzcrw\") pod \"fea86300-bcfd-4c31-893d-342130676629\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") "
Oct 30 19:14:25.294371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294343    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-socket\") pod \"fea86300-bcfd-4c31-893d-342130676629\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") "
Oct 30 19:14:25.294481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294397    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-envoy\") pod \"fea86300-bcfd-4c31-893d-342130676629\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") "
Oct 30 19:14:25.294592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294556    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-credential-socket" pod "fea86300-bcfd-4c31-893d-342130676629" (UID: "fea86300-bcfd-4c31-893d-342130676629"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:25.294592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294576    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-data" pod "fea86300-bcfd-4c31-893d-342130676629" (UID: "fea86300-bcfd-4c31-893d-342130676629"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:25.294779 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294756    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-socket" pod "fea86300-bcfd-4c31-893d-342130676629" (UID: "fea86300-bcfd-4c31-893d-342130676629"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:25.294868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.294846    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea86300-bcfd-4c31-893d-342130676629-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea86300-bcfd-4c31-893d-342130676629-istiod-ca-cert\") pod \"fea86300-bcfd-4c31-893d-342130676629\" (UID: \"fea86300-bcfd-4c31-893d-342130676629\") "
Oct 30 19:14:25.295338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.295308    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea86300-bcfd-4c31-893d-342130676629-istiod-ca-cert" pod "fea86300-bcfd-4c31-893d-342130676629" (UID: "fea86300-bcfd-4c31-893d-342130676629"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:25.295924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.295898    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea86300-bcfd-4c31-893d-342130676629-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.296029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.295926    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.296029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.295943    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.296029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.295956    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.296029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.295971    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.296722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.296700    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-istio-token" pod "fea86300-bcfd-4c31-893d-342130676629" (UID: "fea86300-bcfd-4c31-893d-342130676629"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:25.296816 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.296739    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-kube-api-access-gzcrw" pod "fea86300-bcfd-4c31-893d-342130676629" (UID: "fea86300-bcfd-4c31-893d-342130676629"). InnerVolumeSpecName "kube-api-access-gzcrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:25.296885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.296855    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fea86300-bcfd-4c31-893d-342130676629-istio-podinfo" pod "fea86300-bcfd-4c31-893d-342130676629" (UID: "fea86300-bcfd-4c31-893d-342130676629"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:25.296935 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.296876    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-envoy" pod "fea86300-bcfd-4c31-893d-342130676629" (UID: "fea86300-bcfd-4c31-893d-342130676629"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:25.397651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.397616    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea86300-bcfd-4c31-893d-342130676629-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.397651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.397648    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.397848 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.397658    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzcrw\" (UniqueName: \"kubernetes.io/projected/fea86300-bcfd-4c31-893d-342130676629-kube-api-access-gzcrw\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.397848 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.397669    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea86300-bcfd-4c31-893d-342130676629-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:25.509639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.509542    4267 generic.go:356] "Generic (PLEG): container finished" podID="fea86300-bcfd-4c31-893d-342130676629" containerID="88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc" exitCode=0
Oct 30 19:14:25.509639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.509586    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" event={"ID":"fea86300-bcfd-4c31-893d-342130676629","Type":"ContainerDied","Data":"88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc"}
Oct 30 19:14:25.509639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.509606    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"
Oct 30 19:14:25.509894 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.509648    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj" event={"ID":"fea86300-bcfd-4c31-893d-342130676629","Type":"ContainerDied","Data":"6fac196f0e9f791d4719caee73b18dc1dd347f8cf1b0d092e82459a04ed4f54b"}
Oct 30 19:14:25.509894 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.509672    4267 scope.go:122] "RemoveContainer" containerID="88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc"
Oct 30 19:14:25.524085 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.524063    4267 scope.go:122] "RemoveContainer" containerID="88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc"
Oct 30 19:14:25.524412 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:14:25.524392    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc\": container with ID starting with 88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc not found: ID does not exist" containerID="88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc"
Oct 30 19:14:25.524480 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.524425    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc"} err="failed to get container status \"88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc\": rpc error: code = NotFound desc = could not find container \"88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc\": container with ID starting with 88aa7a12cdecbb9e693e18ad5fe11dd77d2b07a257a9bc11d8fc8666a1a7f7bc not found: ID does not exist"
Oct 30 19:14:25.677265 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.677221    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"]
Oct 30 19:14:25.704922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:25.704847    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-d6nfj"]
Oct 30 19:14:27.224667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:27.224635    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea86300-bcfd-4c31-893d-342130676629" path="/var/lib/kubelet/pods/fea86300-bcfd-4c31-893d-342130676629/volumes"
Oct 30 19:14:28.293296 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:28.293245    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"]
Oct 30 19:14:33.294249 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.294183    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw" podUID="05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9" containerName="payload-pre-processing" containerID="cri-o://f7fac6684d17ea0d4912d64c95d0212279f67938c8ab1a99b69235e10f12d0b6" gracePeriod=25
Oct 30 19:14:33.579226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.579193    4267 generic.go:356] "Generic (PLEG): container finished" podID="05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9" containerID="f7fac6684d17ea0d4912d64c95d0212279f67938c8ab1a99b69235e10f12d0b6" exitCode=0
Oct 30 19:14:33.579351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.579265    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw" event={"ID":"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9","Type":"ContainerDied","Data":"f7fac6684d17ea0d4912d64c95d0212279f67938c8ab1a99b69235e10f12d0b6"}
Oct 30 19:14:33.579351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.579316    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw" event={"ID":"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9","Type":"ContainerDied","Data":"ff8277860ffb24df5f45048444f018d528e65e9864412a1f2357a9b80eb51cc9"}
Oct 30 19:14:33.579351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.579332    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8277860ffb24df5f45048444f018d528e65e9864412a1f2357a9b80eb51cc9"
Oct 30 19:14:33.596203 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.596177    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:14:33.628633 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.628598    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-plugins-config-volume\") pod \"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9\" (UID: \"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9\") "
Oct 30 19:14:33.628844 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.628719    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-kube-api-access-jz9j9\" (UniqueName: \"kubernetes.io/projected/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-kube-api-access-jz9j9\") pod \"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9\" (UID: \"05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9\") "
Oct 30 19:14:33.629050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.629025    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-plugins-config-volume" pod "05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9" (UID: "05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:33.629328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.629305    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:33.631006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.630979    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-kube-api-access-jz9j9" pod "05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9" (UID: "05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9"). InnerVolumeSpecName "kube-api-access-jz9j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:33.731031 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:33.730991    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz9j9\" (UniqueName: \"kubernetes.io/projected/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9-kube-api-access-jz9j9\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:34.586583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:34.586551    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"
Oct 30 19:14:34.751960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:34.751926    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"]
Oct 30 19:14:34.800389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:34.800351    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7d059d-95ff97bff-ztwnw"]
Oct 30 19:14:35.222584 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:35.222552    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9" path="/var/lib/kubelet/pods/05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9/volumes"
Oct 30 19:14:40.911170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911134    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"]
Oct 30 19:14:40.911805 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911789    4267 cpu_manager.go:414] "containerMap: removing container" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy"
Oct 30 19:14:40.911805 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911805    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy"
Oct 30 19:14:40.911919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911818    4267 cpu_manager.go:414] "containerMap: removing container" podUID="fea86300-bcfd-4c31-893d-342130676629" containerName="istio-proxy"
Oct 30 19:14:40.911919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911824    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea86300-bcfd-4c31-893d-342130676629" containerName="istio-proxy"
Oct 30 19:14:40.911919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911853    4267 cpu_manager.go:414] "containerMap: removing container" podUID="05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9" containerName="payload-pre-processing"
Oct 30 19:14:40.911919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911861    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9" containerName="payload-pre-processing"
Oct 30 19:14:40.911919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911875    4267 cpu_manager.go:414] "containerMap: removing container" podUID="defecc52-428f-4f15-acef-aabfb38aa43f" containerName="revoke-keys"
Oct 30 19:14:40.911919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911881    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defecc52-428f-4f15-acef-aabfb38aa43f" containerName="revoke-keys"
Oct 30 19:14:40.912236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911983    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="75951caa-ff7b-4c5b-b55b-11a66a055f17" containerName="istio-proxy"
Oct 30 19:14:40.912236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.911991    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea86300-bcfd-4c31-893d-342130676629" containerName="istio-proxy"
Oct 30 19:14:40.912236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.912006    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f19a3e-b6d1-4ecb-aba1-5497d09d6bc9" containerName="payload-pre-processing"
Oct 30 19:14:40.912236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.912014    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="defecc52-428f-4f15-acef-aabfb38aa43f" containerName="revoke-keys"
Oct 30 19:14:40.915824 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.915805    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:40.967985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.967946    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:40.967985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:40.967964    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:14:41.044226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.037563    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"]
Oct 30 19:14:41.044226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.038416    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:41.044226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.038509    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:41.044226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.038582    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwlds\" (UniqueName: \"kubernetes.io/projected/06c7a105-02f2-496e-ac2d-73f64d42e7bd-kube-api-access-dwlds\") pod \"maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:41.140225 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.140181    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:41.140425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.140247    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:41.140425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.140296    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwlds\" (UniqueName: \"kubernetes.io/projected/06c7a105-02f2-496e-ac2d-73f64d42e7bd-kube-api-access-dwlds\") pod \"maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:41.140862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.140839    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:41.142800 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.142774    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:41.197550 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.197514    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwlds\" (UniqueName: \"kubernetes.io/projected/06c7a105-02f2-496e-ac2d-73f64d42e7bd-kube-api-access-dwlds\") pod \"maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:41.230175 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.230140    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:41.610723 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:14:41.610680    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c7a105_02f2_496e_ac2d_73f64d42e7bd.slice/crio-f23a5db810fa0331963813da4cc8f4d48c9d692dcac970f448ea09fa827236d7 WatchSource:0}: Error finding container f23a5db810fa0331963813da4cc8f4d48c9d692dcac970f448ea09fa827236d7: Status 404 returned error can't find the container with id f23a5db810fa0331963813da4cc8f4d48c9d692dcac970f448ea09fa827236d7
Oct 30 19:14:41.611001 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.610981    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"]
Oct 30 19:14:41.648152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:41.648117    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9" event={"ID":"06c7a105-02f2-496e-ac2d-73f64d42e7bd","Type":"ContainerStarted","Data":"f23a5db810fa0331963813da4cc8f4d48c9d692dcac970f448ea09fa827236d7"}
Oct 30 19:14:42.658834 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:42.658800    4267 generic.go:356] "Generic (PLEG): container finished" podID="06c7a105-02f2-496e-ac2d-73f64d42e7bd" containerID="150cbab8332b37e077451d55bce0e5118d40c00cd011a6cf65efa2a48f0571f8" exitCode=0
Oct 30 19:14:42.659274 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:42.658884    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9" event={"ID":"06c7a105-02f2-496e-ac2d-73f64d42e7bd","Type":"ContainerDied","Data":"150cbab8332b37e077451d55bce0e5118d40c00cd011a6cf65efa2a48f0571f8"}
Oct 30 19:14:43.851775 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:43.851744    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:43.992052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:43.992013    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-openshift-ca\") pod \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") "
Oct 30 19:14:43.992265 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:43.992068    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c7a105-02f2-496e-ac2d-73f64d42e7bd-kube-api-access-dwlds\" (UniqueName: \"kubernetes.io/projected/06c7a105-02f2-496e-ac2d-73f64d42e7bd-kube-api-access-dwlds\") pod \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") "
Oct 30 19:14:43.992347 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:43.992328    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-xks-ca\") pod \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\" (UID: \"06c7a105-02f2-496e-ac2d-73f64d42e7bd\") "
Oct 30 19:14:43.992543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:43.992470    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-openshift-ca" pod "06c7a105-02f2-496e-ac2d-73f64d42e7bd" (UID: "06c7a105-02f2-496e-ac2d-73f64d42e7bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:43.993192 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:43.993162    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:43.994938 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:43.994906    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c7a105-02f2-496e-ac2d-73f64d42e7bd-kube-api-access-dwlds" pod "06c7a105-02f2-496e-ac2d-73f64d42e7bd" (UID: "06c7a105-02f2-496e-ac2d-73f64d42e7bd"). InnerVolumeSpecName "kube-api-access-dwlds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:44.000130 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:44.000096    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-xks-ca" pod "06c7a105-02f2-496e-ac2d-73f64d42e7bd" (UID: "06c7a105-02f2-496e-ac2d-73f64d42e7bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:44.094448 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:44.094399    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwlds\" (UniqueName: \"kubernetes.io/projected/06c7a105-02f2-496e-ac2d-73f64d42e7bd-kube-api-access-dwlds\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:44.094448 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:44.094440    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c7a105-02f2-496e-ac2d-73f64d42e7bd-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:44.676682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:44.676651    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"
Oct 30 19:14:44.676918 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:44.676658    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9" event={"ID":"06c7a105-02f2-496e-ac2d-73f64d42e7bd","Type":"ContainerDied","Data":"f23a5db810fa0331963813da4cc8f4d48c9d692dcac970f448ea09fa827236d7"}
Oct 30 19:14:44.676918 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:44.676759    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f23a5db810fa0331963813da4cc8f4d48c9d692dcac970f448ea09fa827236d7"
Oct 30 19:14:50.159445 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:14:50.159400    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:14:52.090504 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:52.090458    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"]
Oct 30 19:14:52.211695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:52.211641    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"]
Oct 30 19:14:55.207789 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.207752    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"]
Oct 30 19:14:55.208279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.207951    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz" podUID="8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330" containerName="maas-api" containerID="cri-o://a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3" gracePeriod=30
Oct 30 19:14:55.568923 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.568895    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:14:55.608220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.608178    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-maas-api-tls\") pod \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") "
Oct 30 19:14:55.608705 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.608677    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-kube-api-access-qtqjg\" (UniqueName: \"kubernetes.io/projected/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-kube-api-access-qtqjg\") pod \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") "
Oct 30 19:14:55.608940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.608923    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-openshift-service-ca\") pod \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") "
Oct 30 19:14:55.609143 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.609127    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-metrics-certs\") pod \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\" (UID: \"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330\") "
Oct 30 19:14:55.612171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.611897    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-openshift-service-ca" pod "8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330" (UID: "8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:55.618004 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.617971    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-kube-api-access-qtqjg" pod "8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330" (UID: "8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330"). InnerVolumeSpecName "kube-api-access-qtqjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:55.618160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.618140    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-metrics-certs" pod "8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330" (UID: "8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:55.623515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.620109    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-maas-api-tls" pod "8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330" (UID: "8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:55.716936 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.716466    4267 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-openshift-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.716936 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.716618    4267 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-metrics-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.716936 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.716639    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-maas-api-tls\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.716936 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.716656    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtqjg\" (UniqueName: \"kubernetes.io/projected/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330-kube-api-access-qtqjg\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.783593 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.783554    4267 generic.go:356] "Generic (PLEG): container finished" podID="8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330" containerID="a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3" exitCode=0
Oct 30 19:14:55.783804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.783616    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"
Oct 30 19:14:55.783804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.783641    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz" event={"ID":"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330","Type":"ContainerDied","Data":"a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3"}
Oct 30 19:14:55.783804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.783704    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz" event={"ID":"8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330","Type":"ContainerDied","Data":"ff17a6d203f6523544a5b05b4518d6cb5f92b34163b50fb178b373c7c7b4fd73"}
Oct 30 19:14:55.783804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.783719    4267 scope.go:122] "RemoveContainer" containerID="a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3"
Oct 30 19:14:55.796553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.796523    4267 scope.go:122] "RemoveContainer" containerID="a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3"
Oct 30 19:14:55.796940 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:14:55.796921    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3\": container with ID starting with a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3 not found: ID does not exist" containerID="a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3"
Oct 30 19:14:55.797004 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.796949    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3"} err="failed to get container status \"a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3\": rpc error: code = NotFound desc = could not find container \"a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3\": container with ID starting with a33ea3a144c0e7dd06492756df0866125cdf18f9e32db0d1f024419a9d7c3bd3 not found: ID does not exist"
Oct 30 19:14:55.950296 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.950253    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"]
Oct 30 19:14:55.955475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:55.955444    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-322e97-559598dfd9-jdtzz"]
Oct 30 19:14:57.091455 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.091375    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s" podUID="34103503-dc6a-4ca3-8644-e8bb750fc414" containerName="payload-processing" containerID="cri-o://2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46" gracePeriod=25
Oct 30 19:14:57.127722 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:14:57.127691    4267 logging.go:55] [core] [Channel #3969 SubChannel #3970]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.133:50052", ServerName: "10.133.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.133:50052: connect: connection refused"
Oct 30 19:14:57.212441 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.212379    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7" podUID="add9774b-fbc8-4434-a91b-ed2be5fba187" containerName="payload-pre-processing" containerID="cri-o://e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad" gracePeriod=25
Oct 30 19:14:57.226847 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.226818    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330" path="/var/lib/kubelet/pods/8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330/volumes"
Oct 30 19:14:57.420121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.419688    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"]
Oct 30 19:14:57.446853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.446823    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:14:57.460726 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.460697    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-322e97-4r44l"]
Oct 30 19:14:57.545633 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.545604    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34103503-dc6a-4ca3-8644-e8bb750fc414-kube-api-access-2q2kf\" (UniqueName: \"kubernetes.io/projected/34103503-dc6a-4ca3-8644-e8bb750fc414-kube-api-access-2q2kf\") pod \"34103503-dc6a-4ca3-8644-e8bb750fc414\" (UID: \"34103503-dc6a-4ca3-8644-e8bb750fc414\") "
Oct 30 19:14:57.545823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.545729    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34103503-dc6a-4ca3-8644-e8bb750fc414-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34103503-dc6a-4ca3-8644-e8bb750fc414-plugins-config-volume\") pod \"34103503-dc6a-4ca3-8644-e8bb750fc414\" (UID: \"34103503-dc6a-4ca3-8644-e8bb750fc414\") "
Oct 30 19:14:57.546752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.546720    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34103503-dc6a-4ca3-8644-e8bb750fc414-plugins-config-volume" pod "34103503-dc6a-4ca3-8644-e8bb750fc414" (UID: "34103503-dc6a-4ca3-8644-e8bb750fc414"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:57.548460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.548436    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34103503-dc6a-4ca3-8644-e8bb750fc414-kube-api-access-2q2kf" pod "34103503-dc6a-4ca3-8644-e8bb750fc414" (UID: "34103503-dc6a-4ca3-8644-e8bb750fc414"). InnerVolumeSpecName "kube-api-access-2q2kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:57.572744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.572714    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:14:57.646955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.646864    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add9774b-fbc8-4434-a91b-ed2be5fba187-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add9774b-fbc8-4434-a91b-ed2be5fba187-plugins-config-volume\") pod \"add9774b-fbc8-4434-a91b-ed2be5fba187\" (UID: \"add9774b-fbc8-4434-a91b-ed2be5fba187\") "
Oct 30 19:14:57.647135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.647033    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add9774b-fbc8-4434-a91b-ed2be5fba187-kube-api-access-hphh5\" (UniqueName: \"kubernetes.io/projected/add9774b-fbc8-4434-a91b-ed2be5fba187-kube-api-access-hphh5\") pod \"add9774b-fbc8-4434-a91b-ed2be5fba187\" (UID: \"add9774b-fbc8-4434-a91b-ed2be5fba187\") "
Oct 30 19:14:57.647338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.647309    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add9774b-fbc8-4434-a91b-ed2be5fba187-plugins-config-volume" pod "add9774b-fbc8-4434-a91b-ed2be5fba187" (UID: "add9774b-fbc8-4434-a91b-ed2be5fba187"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:57.647720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.647694    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/add9774b-fbc8-4434-a91b-ed2be5fba187-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:57.647720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.647719    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q2kf\" (UniqueName: \"kubernetes.io/projected/34103503-dc6a-4ca3-8644-e8bb750fc414-kube-api-access-2q2kf\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:57.647916 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.647734    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34103503-dc6a-4ca3-8644-e8bb750fc414-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:57.649800 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.649780    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add9774b-fbc8-4434-a91b-ed2be5fba187-kube-api-access-hphh5" pod "add9774b-fbc8-4434-a91b-ed2be5fba187" (UID: "add9774b-fbc8-4434-a91b-ed2be5fba187"). InnerVolumeSpecName "kube-api-access-hphh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:57.749610 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.749569    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hphh5\" (UniqueName: \"kubernetes.io/projected/add9774b-fbc8-4434-a91b-ed2be5fba187-kube-api-access-hphh5\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:14:57.803800 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.803766    4267 generic.go:356] "Generic (PLEG): container finished" podID="add9774b-fbc8-4434-a91b-ed2be5fba187" containerID="e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad" exitCode=0
Oct 30 19:14:57.803987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.803814    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"
Oct 30 19:14:57.803987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.803858    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7" event={"ID":"add9774b-fbc8-4434-a91b-ed2be5fba187","Type":"ContainerDied","Data":"e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad"}
Oct 30 19:14:57.803987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.803921    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7" event={"ID":"add9774b-fbc8-4434-a91b-ed2be5fba187","Type":"ContainerDied","Data":"80445e402ce9dc3a1b416f9fafc1b4df6b1d7c359b1455d75fd4f7d167aca7fc"}
Oct 30 19:14:57.803987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.803952    4267 scope.go:122] "RemoveContainer" containerID="e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad"
Oct 30 19:14:57.805449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.805425    4267 generic.go:356] "Generic (PLEG): container finished" podID="34103503-dc6a-4ca3-8644-e8bb750fc414" containerID="2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46" exitCode=0
Oct 30 19:14:57.805606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.805462    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"
Oct 30 19:14:57.805606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.805522    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s" event={"ID":"34103503-dc6a-4ca3-8644-e8bb750fc414","Type":"ContainerDied","Data":"2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46"}
Oct 30 19:14:57.805606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.805583    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s" event={"ID":"34103503-dc6a-4ca3-8644-e8bb750fc414","Type":"ContainerDied","Data":"d732ec1a38dbd02e0897f47e52f57c93247b25ebd24aae28e2c15052dc7ac8aa"}
Oct 30 19:14:57.823133 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.823023    4267 scope.go:122] "RemoveContainer" containerID="e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad"
Oct 30 19:14:57.823949 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:14:57.823924    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad\": container with ID starting with e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad not found: ID does not exist" containerID="e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad"
Oct 30 19:14:57.824040 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.823958    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad"} err="failed to get container status \"e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad\": rpc error: code = NotFound desc = could not find container \"e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad\": container with ID starting with e42bbaa7b283b31c2da161bd3b50a06f7d0cbe03ecc0ad2b1f22eb81b2ab1aad not found: ID does not exist"
Oct 30 19:14:57.824040 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.823981    4267 scope.go:122] "RemoveContainer" containerID="2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46"
Oct 30 19:14:57.834791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.834764    4267 scope.go:122] "RemoveContainer" containerID="2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46"
Oct 30 19:14:57.835096 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:14:57.835076    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46\": container with ID starting with 2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46 not found: ID does not exist" containerID="2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46"
Oct 30 19:14:57.835158 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.835109    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46"} err="failed to get container status \"2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46\": rpc error: code = NotFound desc = could not find container \"2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46\": container with ID starting with 2d5085bb03529bb54f1ea5fbc2fa8362b98310e259f81d689ffa155a88fb4e46 not found: ID does not exist"
Oct 30 19:14:57.970838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.970805    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"]
Oct 30 19:14:57.975080 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:57.975049    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f26b1d-6d7744998c-zt5p7"]
Oct 30 19:14:58.081315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:58.081282    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"]
Oct 30 19:14:58.085086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:58.085062    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s"]
Oct 30 19:14:58.128166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:58.128132    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-f26b1d-5f5fcfbf7c-25w2s" podUID="34103503-dc6a-4ca3-8644-e8bb750fc414" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.133:50052\" within 1s: context deadline exceeded"
Oct 30 19:14:59.226455 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:59.226416    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34103503-dc6a-4ca3-8644-e8bb750fc414" path="/var/lib/kubelet/pods/34103503-dc6a-4ca3-8644-e8bb750fc414/volumes"
Oct 30 19:14:59.227032 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:59.227014    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add9774b-fbc8-4434-a91b-ed2be5fba187" path="/var/lib/kubelet/pods/add9774b-fbc8-4434-a91b-ed2be5fba187/volumes"
Oct 30 19:14:59.227551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:14:59.227524    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf4ac3e-7f60-4006-8346-f95ad7a51a48" path="/var/lib/kubelet/pods/fbf4ac3e-7f60-4006-8346-f95ad7a51a48/volumes"
Oct 30 19:15:00.166808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.166773    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"]
Oct 30 19:15:00.167466 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167450    4267 cpu_manager.go:414] "containerMap: removing container" podUID="34103503-dc6a-4ca3-8644-e8bb750fc414" containerName="payload-processing"
Oct 30 19:15:00.167543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167467    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34103503-dc6a-4ca3-8644-e8bb750fc414" containerName="payload-processing"
Oct 30 19:15:00.167543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167479    4267 cpu_manager.go:414] "containerMap: removing container" podUID="06c7a105-02f2-496e-ac2d-73f64d42e7bd" containerName="revoke-keys"
Oct 30 19:15:00.167543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167486    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c7a105-02f2-496e-ac2d-73f64d42e7bd" containerName="revoke-keys"
Oct 30 19:15:00.167543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167529    4267 cpu_manager.go:414] "containerMap: removing container" podUID="add9774b-fbc8-4434-a91b-ed2be5fba187" containerName="payload-pre-processing"
Oct 30 19:15:00.167543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167535    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add9774b-fbc8-4434-a91b-ed2be5fba187" containerName="payload-pre-processing"
Oct 30 19:15:00.167698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167560    4267 cpu_manager.go:414] "containerMap: removing container" podUID="8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330" containerName="maas-api"
Oct 30 19:15:00.167698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167565    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330" containerName="maas-api"
Oct 30 19:15:00.167698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167674    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="add9774b-fbc8-4434-a91b-ed2be5fba187" containerName="payload-pre-processing"
Oct 30 19:15:00.167698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167685    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="8000ef1b-4de0-4a3a-ba78-a4f5aaa6c330" containerName="maas-api"
Oct 30 19:15:00.167698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167695    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="34103503-dc6a-4ca3-8644-e8bb750fc414" containerName="payload-processing"
Oct 30 19:15:00.167841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.167702    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c7a105-02f2-496e-ac2d-73f64d42e7bd" containerName="revoke-keys"
Oct 30 19:15:00.172871 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.172849    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"
Oct 30 19:15:00.237252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.235045    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"]
Oct 30 19:15:00.240149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.240125    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"
Oct 30 19:15:00.242572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.242548    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"]
Oct 30 19:15:00.247234 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.247164    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"
Oct 30 19:15:00.250132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.250101    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"]
Oct 30 19:15:00.257984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.257940    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:15:00.260405 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.260245    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"]
Oct 30 19:15:00.260663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.260360    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"
Oct 30 19:15:00.265778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.265690    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"]
Oct 30 19:15:00.268189 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.265933    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"
Oct 30 19:15:00.270444 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.270423    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"
Oct 30 19:15:00.270641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.270616    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"]
Oct 30 19:15:00.277271 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.276420    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"
Oct 30 19:15:00.279060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.279019    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"]
Oct 30 19:15:00.288830 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.288807    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"]
Oct 30 19:15:00.289073 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.288903    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"
Oct 30 19:15:00.291050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.291024    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxstk\" (UniqueName: \"kubernetes.io/projected/b98b2065-c604-4e53-b671-94573cad2963-kube-api-access-fxstk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc\" (UID: \"b98b2065-c604-4e53-b671-94573cad2963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"
Oct 30 19:15:00.291366 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.291341    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bb6f2b-8772-4579-88d8-4a347aeee899-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q\" (UID: \"48bb6f2b-8772-4579-88d8-4a347aeee899\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"
Oct 30 19:15:00.291576 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.291555    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecaf59c7-05b4-4787-8d74-fb373c68838b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp\" (UID: \"ecaf59c7-05b4-4787-8d74-fb373c68838b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"
Oct 30 19:15:00.291691 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.291671    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98b2065-c604-4e53-b671-94573cad2963-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc\" (UID: \"b98b2065-c604-4e53-b671-94573cad2963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"
Oct 30 19:15:00.291756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.291715    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76c6s\" (UniqueName: \"kubernetes.io/projected/ecaf59c7-05b4-4787-8d74-fb373c68838b-kube-api-access-76c6s\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp\" (UID: \"ecaf59c7-05b4-4787-8d74-fb373c68838b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"
Oct 30 19:15:00.292922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.291809    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99rzn\" (UniqueName: \"kubernetes.io/projected/48bb6f2b-8772-4579-88d8-4a347aeee899-kube-api-access-99rzn\") pod \"maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q\" (UID: \"48bb6f2b-8772-4579-88d8-4a347aeee899\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"
Oct 30 19:15:00.297591 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.297566    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"]
Oct 30 19:15:00.335757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.304744    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"]
Oct 30 19:15:00.335757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.335107    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"]
Oct 30 19:15:00.335757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.335182    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"]
Oct 30 19:15:00.335757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.335318    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"
Oct 30 19:15:00.372031 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.371922    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"]
Oct 30 19:15:00.377895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.377864    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"]
Oct 30 19:15:00.383390 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.383353    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"]
Oct 30 19:15:00.388028 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.388003    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"]
Oct 30 19:15:00.392755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.392711    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"]
Oct 30 19:15:00.393353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.393327    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842cbd49-4cce-4500-b92b-d9b1446926c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5\" (UID: \"842cbd49-4cce-4500-b92b-d9b1446926c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"
Oct 30 19:15:00.393537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.393518    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bb6f2b-8772-4579-88d8-4a347aeee899-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q\" (UID: \"48bb6f2b-8772-4579-88d8-4a347aeee899\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"
Oct 30 19:15:00.393774 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.393752    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6dtb\" (UniqueName: \"kubernetes.io/projected/842cbd49-4cce-4500-b92b-d9b1446926c2-kube-api-access-q6dtb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5\" (UID: \"842cbd49-4cce-4500-b92b-d9b1446926c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"
Oct 30 19:15:00.393984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.393966    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4qg2\" (UniqueName: \"kubernetes.io/projected/0b63f080-3d3f-46ec-983c-5fe41070c214-kube-api-access-r4qg2\") pod \"maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq\" (UID: \"0b63f080-3d3f-46ec-983c-5fe41070c214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"
Oct 30 19:15:00.394173 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.394156    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecaf59c7-05b4-4787-8d74-fb373c68838b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp\" (UID: \"ecaf59c7-05b4-4787-8d74-fb373c68838b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"
Oct 30 19:15:00.395081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.394325    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bb6f2b-8772-4579-88d8-4a347aeee899-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q\" (UID: \"48bb6f2b-8772-4579-88d8-4a347aeee899\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"
Oct 30 19:15:00.395212 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.394974    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecaf59c7-05b4-4787-8d74-fb373c68838b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp\" (UID: \"ecaf59c7-05b4-4787-8d74-fb373c68838b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"
Oct 30 19:15:00.395386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.395367    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7lk\" (UniqueName: \"kubernetes.io/projected/31bdf1c9-20f7-449c-8807-88c5fbe86b35-kube-api-access-mq7lk\") pod \"maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk\" (UID: \"31bdf1c9-20f7-449c-8807-88c5fbe86b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"
Oct 30 19:15:00.395583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.395564    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c1e641-75e1-4302-82e0-da4f6205f3d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z\" (UID: \"d8c1e641-75e1-4302-82e0-da4f6205f3d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"
Oct 30 19:15:00.396583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.395749    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98b2065-c604-4e53-b671-94573cad2963-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc\" (UID: \"b98b2065-c604-4e53-b671-94573cad2963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"
Oct 30 19:15:00.396704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.396607    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7\" (UID: \"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"
Oct 30 19:15:00.396704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.396647    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76c6s\" (UniqueName: \"kubernetes.io/projected/ecaf59c7-05b4-4787-8d74-fb373c68838b-kube-api-access-76c6s\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp\" (UID: \"ecaf59c7-05b4-4787-8d74-fb373c68838b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"
Oct 30 19:15:00.396704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.396687    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1c3881-68a5-4020-adf3-a7399af81b17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw\" (UID: \"0e1c3881-68a5-4020-adf3-a7399af81b17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"
Oct 30 19:15:00.396880 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.396716    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bdf1c9-20f7-449c-8807-88c5fbe86b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk\" (UID: \"31bdf1c9-20f7-449c-8807-88c5fbe86b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"
Oct 30 19:15:00.396880 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.396773    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8lr\" (UniqueName: \"kubernetes.io/projected/d8c1e641-75e1-4302-82e0-da4f6205f3d8-kube-api-access-wd8lr\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z\" (UID: \"d8c1e641-75e1-4302-82e0-da4f6205f3d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"
Oct 30 19:15:00.396880 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.396816    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99rzn\" (UniqueName: \"kubernetes.io/projected/48bb6f2b-8772-4579-88d8-4a347aeee899-kube-api-access-99rzn\") pod \"maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q\" (UID: \"48bb6f2b-8772-4579-88d8-4a347aeee899\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"
Oct 30 19:15:00.397048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.396905    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b63f080-3d3f-46ec-983c-5fe41070c214-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq\" (UID: \"0b63f080-3d3f-46ec-983c-5fe41070c214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"
Oct 30 19:15:00.397048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.396947    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl4tc\" (UniqueName: \"kubernetes.io/projected/0e1c3881-68a5-4020-adf3-a7399af81b17-kube-api-access-vl4tc\") pod \"maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw\" (UID: \"0e1c3881-68a5-4020-adf3-a7399af81b17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"
Oct 30 19:15:00.397048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.396976    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshtb\" (UniqueName: \"kubernetes.io/projected/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-kube-api-access-cshtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7\" (UID: \"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"
Oct 30 19:15:00.397048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.397025    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxstk\" (UniqueName: \"kubernetes.io/projected/b98b2065-c604-4e53-b671-94573cad2963-kube-api-access-fxstk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc\" (UID: \"b98b2065-c604-4e53-b671-94573cad2963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"
Oct 30 19:15:00.397451 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.396553    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98b2065-c604-4e53-b671-94573cad2963-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc\" (UID: \"b98b2065-c604-4e53-b671-94573cad2963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"
Oct 30 19:15:00.491295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.491230    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76c6s\" (UniqueName: \"kubernetes.io/projected/ecaf59c7-05b4-4787-8d74-fb373c68838b-kube-api-access-76c6s\") pod \"maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp\" (UID: \"ecaf59c7-05b4-4787-8d74-fb373c68838b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"
Oct 30 19:15:00.497253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.497207    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99rzn\" (UniqueName: \"kubernetes.io/projected/48bb6f2b-8772-4579-88d8-4a347aeee899-kube-api-access-99rzn\") pod \"maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q\" (UID: \"48bb6f2b-8772-4579-88d8-4a347aeee899\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"
Oct 30 19:15:00.500147 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.500111    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b63f080-3d3f-46ec-983c-5fe41070c214-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq\" (UID: \"0b63f080-3d3f-46ec-983c-5fe41070c214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"
Oct 30 19:15:00.500362 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.500347    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl4tc\" (UniqueName: \"kubernetes.io/projected/0e1c3881-68a5-4020-adf3-a7399af81b17-kube-api-access-vl4tc\") pod \"maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw\" (UID: \"0e1c3881-68a5-4020-adf3-a7399af81b17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"
Oct 30 19:15:00.500473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.500458    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshtb\" (UniqueName: \"kubernetes.io/projected/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-kube-api-access-cshtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7\" (UID: \"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"
Oct 30 19:15:00.500714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.500697    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842cbd49-4cce-4500-b92b-d9b1446926c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5\" (UID: \"842cbd49-4cce-4500-b92b-d9b1446926c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"
Oct 30 19:15:00.500861 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.500847    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6dtb\" (UniqueName: \"kubernetes.io/projected/842cbd49-4cce-4500-b92b-d9b1446926c2-kube-api-access-q6dtb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5\" (UID: \"842cbd49-4cce-4500-b92b-d9b1446926c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"
Oct 30 19:15:00.501001 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.500987    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4qg2\" (UniqueName: \"kubernetes.io/projected/0b63f080-3d3f-46ec-983c-5fe41070c214-kube-api-access-r4qg2\") pod \"maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq\" (UID: \"0b63f080-3d3f-46ec-983c-5fe41070c214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"
Oct 30 19:15:00.501170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.501155    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7lk\" (UniqueName: \"kubernetes.io/projected/31bdf1c9-20f7-449c-8807-88c5fbe86b35-kube-api-access-mq7lk\") pod \"maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk\" (UID: \"31bdf1c9-20f7-449c-8807-88c5fbe86b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"
Oct 30 19:15:00.501270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.501256    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c1e641-75e1-4302-82e0-da4f6205f3d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z\" (UID: \"d8c1e641-75e1-4302-82e0-da4f6205f3d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"
Oct 30 19:15:00.501382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.501369    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7\" (UID: \"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"
Oct 30 19:15:00.501512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.501482    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1c3881-68a5-4020-adf3-a7399af81b17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw\" (UID: \"0e1c3881-68a5-4020-adf3-a7399af81b17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"
Oct 30 19:15:00.501633 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.501617    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bdf1c9-20f7-449c-8807-88c5fbe86b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk\" (UID: \"31bdf1c9-20f7-449c-8807-88c5fbe86b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"
Oct 30 19:15:00.501765 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.501748    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8lr\" (UniqueName: \"kubernetes.io/projected/d8c1e641-75e1-4302-82e0-da4f6205f3d8-kube-api-access-wd8lr\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z\" (UID: \"d8c1e641-75e1-4302-82e0-da4f6205f3d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"
Oct 30 19:15:00.503983 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.503958    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b63f080-3d3f-46ec-983c-5fe41070c214-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq\" (UID: \"0b63f080-3d3f-46ec-983c-5fe41070c214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"
Oct 30 19:15:00.511348 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.505446    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842cbd49-4cce-4500-b92b-d9b1446926c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5\" (UID: \"842cbd49-4cce-4500-b92b-d9b1446926c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"
Oct 30 19:15:00.511564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.507179    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bdf1c9-20f7-449c-8807-88c5fbe86b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk\" (UID: \"31bdf1c9-20f7-449c-8807-88c5fbe86b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"
Oct 30 19:15:00.511697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.509808    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c1e641-75e1-4302-82e0-da4f6205f3d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z\" (UID: \"d8c1e641-75e1-4302-82e0-da4f6205f3d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"
Oct 30 19:15:00.511812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.510600    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7\" (UID: \"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"
Oct 30 19:15:00.511948 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.511223    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1c3881-68a5-4020-adf3-a7399af81b17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw\" (UID: \"0e1c3881-68a5-4020-adf3-a7399af81b17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"
Oct 30 19:15:00.531552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.517192    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxstk\" (UniqueName: \"kubernetes.io/projected/b98b2065-c604-4e53-b671-94573cad2963-kube-api-access-fxstk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc\" (UID: \"b98b2065-c604-4e53-b671-94573cad2963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"
Oct 30 19:15:00.562713 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.562556    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"
Oct 30 19:15:00.563529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.562873    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshtb\" (UniqueName: \"kubernetes.io/projected/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-kube-api-access-cshtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7\" (UID: \"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"
Oct 30 19:15:00.563529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.563079    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl4tc\" (UniqueName: \"kubernetes.io/projected/0e1c3881-68a5-4020-adf3-a7399af81b17-kube-api-access-vl4tc\") pod \"maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw\" (UID: \"0e1c3881-68a5-4020-adf3-a7399af81b17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"
Oct 30 19:15:00.563661 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.563557    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6dtb\" (UniqueName: \"kubernetes.io/projected/842cbd49-4cce-4500-b92b-d9b1446926c2-kube-api-access-q6dtb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5\" (UID: \"842cbd49-4cce-4500-b92b-d9b1446926c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"
Oct 30 19:15:00.564180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.564155    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4qg2\" (UniqueName: \"kubernetes.io/projected/0b63f080-3d3f-46ec-983c-5fe41070c214-kube-api-access-r4qg2\") pod \"maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq\" (UID: \"0b63f080-3d3f-46ec-983c-5fe41070c214\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"
Oct 30 19:15:00.565470 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.565448    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8lr\" (UniqueName: \"kubernetes.io/projected/d8c1e641-75e1-4302-82e0-da4f6205f3d8-kube-api-access-wd8lr\") pod \"maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z\" (UID: \"d8c1e641-75e1-4302-82e0-da4f6205f3d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"
Oct 30 19:15:00.574059 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.574030    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7lk\" (UniqueName: \"kubernetes.io/projected/31bdf1c9-20f7-449c-8807-88c5fbe86b35-kube-api-access-mq7lk\") pod \"maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk\" (UID: \"31bdf1c9-20f7-449c-8807-88c5fbe86b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"
Oct 30 19:15:00.575770 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.575746    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"
Oct 30 19:15:00.594469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.594443    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"
Oct 30 19:15:00.605072 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.605042    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"
Oct 30 19:15:00.643873 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.641845    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"
Oct 30 19:15:00.643873 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.642776    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"
Oct 30 19:15:00.653798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.652574    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"
Oct 30 19:15:00.658594 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.658544    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"
Oct 30 19:15:00.794863 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.794618    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"
Oct 30 19:15:00.985710 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:00.975574    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"]
Oct 30 19:15:01.116649 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.116619    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"]
Oct 30 19:15:01.140245 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.140219    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"]
Oct 30 19:15:01.236005 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.235981    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"]
Oct 30 19:15:01.236502 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.236471    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"]
Oct 30 19:15:01.239469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.239445    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"]
Oct 30 19:15:01.274932 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.274902    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"]
Oct 30 19:15:01.279652 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:15:01.279620    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bb6f2b_8772_4579_88d8_4a347aeee899.slice/crio-b8af50729e204e3228736e28d953e81beeb8c6c35de40913151f92ae452a5911 WatchSource:0}: Error finding container b8af50729e204e3228736e28d953e81beeb8c6c35de40913151f92ae452a5911: Status 404 returned error can't find the container with id b8af50729e204e3228736e28d953e81beeb8c6c35de40913151f92ae452a5911
Oct 30 19:15:01.367980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.367948    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"]
Oct 30 19:15:01.371863 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.371839    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"]
Oct 30 19:15:01.867675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.867635    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" event={"ID":"b98b2065-c604-4e53-b671-94573cad2963","Type":"ContainerStarted","Data":"141958baf473039129799b4c4ff9c37e1fa01c3d63f678399a2a9dc47089934c"}
Oct 30 19:15:01.867858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.867700    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" event={"ID":"b98b2065-c604-4e53-b671-94573cad2963","Type":"ContainerStarted","Data":"64a224ccdafb42ef205ee8be191aa5d2e1c9c1f3de9c357179bbbf674a462691"}
Oct 30 19:15:01.869011 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.868986    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" event={"ID":"48bb6f2b-8772-4579-88d8-4a347aeee899","Type":"ContainerStarted","Data":"b8af50729e204e3228736e28d953e81beeb8c6c35de40913151f92ae452a5911"}
Oct 30 19:15:01.870514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.870478    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" event={"ID":"31bdf1c9-20f7-449c-8807-88c5fbe86b35","Type":"ContainerStarted","Data":"a216d0c9c3a4dffdd0e331e3d9dd8efe7b58d8f24627c5e8c41fe4fe0e69387b"}
Oct 30 19:15:01.870613 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.870529    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" event={"ID":"31bdf1c9-20f7-449c-8807-88c5fbe86b35","Type":"ContainerStarted","Data":"c0474b33b337e5b014a62b3e1005d4ea1aec5af9ab4fc5b7403359a31fd3eb74"}
Oct 30 19:15:01.871664 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.871644    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" event={"ID":"842cbd49-4cce-4500-b92b-d9b1446926c2","Type":"ContainerStarted","Data":"d31ef08d93e81ecc31134ca622f3c398bae44a5156c265253695095d1e0ad20e"}
Oct 30 19:15:01.873060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.873038    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" event={"ID":"ecaf59c7-05b4-4787-8d74-fb373c68838b","Type":"ContainerStarted","Data":"de0bea6141cb33b62e7856755bf159914a54ed1540c2e2ef9f9dfda4488d1cac"}
Oct 30 19:15:01.873144 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.873083    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" event={"ID":"ecaf59c7-05b4-4787-8d74-fb373c68838b","Type":"ContainerStarted","Data":"8b9fb44d70e68395099c40d09a2c1a0e9f7761233242fcc470172467fb999e4a"}
Oct 30 19:15:01.874655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.874631    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" event={"ID":"0b63f080-3d3f-46ec-983c-5fe41070c214","Type":"ContainerStarted","Data":"32d4efbea1b441435093a92f3d452137d0fd432262780ed577fdc03e00907a43"}
Oct 30 19:15:01.874729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.874669    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" event={"ID":"0b63f080-3d3f-46ec-983c-5fe41070c214","Type":"ContainerStarted","Data":"56c35c742c907eccc475381d5c81b8a9065ad0ebee70d05bf1baa82bd9772145"}
Oct 30 19:15:01.876150 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.876123    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" event={"ID":"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf","Type":"ContainerStarted","Data":"20533b65057d48cf613e3cef19ef30dcb14ba3a5a239be1ca27c82b6ab262807"}
Oct 30 19:15:01.876242 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.876168    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" event={"ID":"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf","Type":"ContainerStarted","Data":"a41400e0d6243559f241f9bb79b0e25fe7a83eff8ffdf01f0e8300b213da6790"}
Oct 30 19:15:01.877664 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.877641    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" event={"ID":"0e1c3881-68a5-4020-adf3-a7399af81b17","Type":"ContainerStarted","Data":"885083ee0064d6c09f00f70e451c87bdd41aaf8ee9744fd7c544d24b589e2c30"}
Oct 30 19:15:01.877748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.877684    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" event={"ID":"0e1c3881-68a5-4020-adf3-a7399af81b17","Type":"ContainerStarted","Data":"344a49be1c468febd3a79d643b4e14bf4dfeaffe45440a2b38b2103a64c28435"}
Oct 30 19:15:01.878897 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.878878    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" event={"ID":"d8c1e641-75e1-4302-82e0-da4f6205f3d8","Type":"ContainerStarted","Data":"ffc2fb8953a2c0746c6494af448df0e82f2e9ac3a03e9bda0ea0315af529f98d"}
Oct 30 19:15:01.985921 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:01.985870    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" podStartSLOduration=1.98585469 podStartE2EDuration="1.98585469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:01.980812973 +0000 UTC m=+2192.868730062" watchObservedRunningTime="2026-10-30 19:15:01.98585469 +0000 UTC m=+2192.873771793"
Oct 30 19:15:02.101428 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.101322    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" podStartSLOduration=2.10130676 podStartE2EDuration="2.10130676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.097579664 +0000 UTC m=+2192.985496754" watchObservedRunningTime="2026-10-30 19:15:02.10130676 +0000 UTC m=+2192.989223849"
Oct 30 19:15:02.219899 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.219837    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" podStartSLOduration=2.219819577 podStartE2EDuration="2.219819577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.215741341 +0000 UTC m=+2193.103658442" watchObservedRunningTime="2026-10-30 19:15:02.219819577 +0000 UTC m=+2193.107736668"
Oct 30 19:15:02.330392 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.330324    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" podStartSLOduration=2.330304163 podStartE2EDuration="2.330304163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.328445626 +0000 UTC m=+2193.216362716" watchObservedRunningTime="2026-10-30 19:15:02.330304163 +0000 UTC m=+2193.218221254"
Oct 30 19:15:02.453304 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.453239    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" podStartSLOduration=2.453219215 podStartE2EDuration="2.453219215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.445279796 +0000 UTC m=+2193.333196886" watchObservedRunningTime="2026-10-30 19:15:02.453219215 +0000 UTC m=+2193.341136305"
Oct 30 19:15:02.557127 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.557075    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" podStartSLOduration=2.55705822 podStartE2EDuration="2.55705822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.556858088 +0000 UTC m=+2193.444775178" watchObservedRunningTime="2026-10-30 19:15:02.55705822 +0000 UTC m=+2193.444975312"
Oct 30 19:15:02.889003 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.888957    4267 generic.go:356] "Generic (PLEG): container finished" podID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerID="1b04dbcbc4639b0cdda6e95e2e986ebebb262927b9841c9d5ea12f26a326006f" exitCode=60
Oct 30 19:15:02.889162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.889050    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" event={"ID":"d8c1e641-75e1-4302-82e0-da4f6205f3d8","Type":"ContainerDied","Data":"1b04dbcbc4639b0cdda6e95e2e986ebebb262927b9841c9d5ea12f26a326006f"}
Oct 30 19:15:02.889378 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.889356    4267 scope.go:122] "RemoveContainer" containerID="1b04dbcbc4639b0cdda6e95e2e986ebebb262927b9841c9d5ea12f26a326006f"
Oct 30 19:15:02.890839 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.890816    4267 generic.go:356] "Generic (PLEG): container finished" podID="b98b2065-c604-4e53-b671-94573cad2963" containerID="141958baf473039129799b4c4ff9c37e1fa01c3d63f678399a2a9dc47089934c" exitCode=60
Oct 30 19:15:02.890934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.890895    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" event={"ID":"b98b2065-c604-4e53-b671-94573cad2963","Type":"ContainerDied","Data":"141958baf473039129799b4c4ff9c37e1fa01c3d63f678399a2a9dc47089934c"}
Oct 30 19:15:02.891150 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.891134    4267 scope.go:122] "RemoveContainer" containerID="141958baf473039129799b4c4ff9c37e1fa01c3d63f678399a2a9dc47089934c"
Oct 30 19:15:02.892629 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.892606    4267 generic.go:356] "Generic (PLEG): container finished" podID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerID="3d4b1b27fe54861d51cdbac7839ab3df92d858144e7ddbe14b4515a372752712" exitCode=60
Oct 30 19:15:02.892734 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.892633    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" event={"ID":"48bb6f2b-8772-4579-88d8-4a347aeee899","Type":"ContainerDied","Data":"3d4b1b27fe54861d51cdbac7839ab3df92d858144e7ddbe14b4515a372752712"}
Oct 30 19:15:02.892950 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.892933    4267 scope.go:122] "RemoveContainer" containerID="3d4b1b27fe54861d51cdbac7839ab3df92d858144e7ddbe14b4515a372752712"
Oct 30 19:15:02.894802 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.894780    4267 generic.go:356] "Generic (PLEG): container finished" podID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerID="a216d0c9c3a4dffdd0e331e3d9dd8efe7b58d8f24627c5e8c41fe4fe0e69387b" exitCode=7
Oct 30 19:15:02.894891 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.894808    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" event={"ID":"31bdf1c9-20f7-449c-8807-88c5fbe86b35","Type":"ContainerDied","Data":"a216d0c9c3a4dffdd0e331e3d9dd8efe7b58d8f24627c5e8c41fe4fe0e69387b"}
Oct 30 19:15:02.895088 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.895068    4267 scope.go:122] "RemoveContainer" containerID="a216d0c9c3a4dffdd0e331e3d9dd8efe7b58d8f24627c5e8c41fe4fe0e69387b"
Oct 30 19:15:02.896524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.896503    4267 generic.go:356] "Generic (PLEG): container finished" podID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerID="3bd538c19ddd51078a165893de01b682b62b12cad78f997099c7e90d6e092608" exitCode=60
Oct 30 19:15:02.896598 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.896535    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" event={"ID":"842cbd49-4cce-4500-b92b-d9b1446926c2","Type":"ContainerDied","Data":"3bd538c19ddd51078a165893de01b682b62b12cad78f997099c7e90d6e092608"}
Oct 30 19:15:02.896818 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.896800    4267 scope.go:122] "RemoveContainer" containerID="3bd538c19ddd51078a165893de01b682b62b12cad78f997099c7e90d6e092608"
Oct 30 19:15:02.898190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.898171    4267 generic.go:356] "Generic (PLEG): container finished" podID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerID="de0bea6141cb33b62e7856755bf159914a54ed1540c2e2ef9f9dfda4488d1cac" exitCode=60
Oct 30 19:15:02.898293 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.898236    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" event={"ID":"ecaf59c7-05b4-4787-8d74-fb373c68838b","Type":"ContainerDied","Data":"de0bea6141cb33b62e7856755bf159914a54ed1540c2e2ef9f9dfda4488d1cac"}
Oct 30 19:15:02.898451 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.898433    4267 scope.go:122] "RemoveContainer" containerID="de0bea6141cb33b62e7856755bf159914a54ed1540c2e2ef9f9dfda4488d1cac"
Oct 30 19:15:02.899779 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.899761    4267 generic.go:356] "Generic (PLEG): container finished" podID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerID="32d4efbea1b441435093a92f3d452137d0fd432262780ed577fdc03e00907a43" exitCode=60
Oct 30 19:15:02.899858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.899781    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" event={"ID":"0b63f080-3d3f-46ec-983c-5fe41070c214","Type":"ContainerDied","Data":"32d4efbea1b441435093a92f3d452137d0fd432262780ed577fdc03e00907a43"}
Oct 30 19:15:02.900073 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.900056    4267 scope.go:122] "RemoveContainer" containerID="32d4efbea1b441435093a92f3d452137d0fd432262780ed577fdc03e00907a43"
Oct 30 19:15:02.901372 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.901349    4267 generic.go:356] "Generic (PLEG): container finished" podID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerID="20533b65057d48cf613e3cef19ef30dcb14ba3a5a239be1ca27c82b6ab262807" exitCode=60
Oct 30 19:15:02.901472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.901447    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" event={"ID":"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf","Type":"ContainerDied","Data":"20533b65057d48cf613e3cef19ef30dcb14ba3a5a239be1ca27c82b6ab262807"}
Oct 30 19:15:02.902642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.902166    4267 scope.go:122] "RemoveContainer" containerID="20533b65057d48cf613e3cef19ef30dcb14ba3a5a239be1ca27c82b6ab262807"
Oct 30 19:15:02.904019 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.904000    4267 generic.go:356] "Generic (PLEG): container finished" podID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerID="885083ee0064d6c09f00f70e451c87bdd41aaf8ee9744fd7c544d24b589e2c30" exitCode=60
Oct 30 19:15:02.904120 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.904032    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" event={"ID":"0e1c3881-68a5-4020-adf3-a7399af81b17","Type":"ContainerDied","Data":"885083ee0064d6c09f00f70e451c87bdd41aaf8ee9744fd7c544d24b589e2c30"}
Oct 30 19:15:02.904320 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:02.904303    4267 scope.go:122] "RemoveContainer" containerID="885083ee0064d6c09f00f70e451c87bdd41aaf8ee9744fd7c544d24b589e2c30"
Oct 30 19:15:03.925150 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.924844    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" event={"ID":"842cbd49-4cce-4500-b92b-d9b1446926c2","Type":"ContainerStarted","Data":"f4530d361159139697fd6188be8ab0fd399738d6c4c999a39f2e4bf2a9171414"}
Oct 30 19:15:03.927239 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.927212    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" event={"ID":"ecaf59c7-05b4-4787-8d74-fb373c68838b","Type":"ContainerStarted","Data":"96e84f071f1eba2f1436324339c0f6cc577a128223ac5f8329cbc504e4651315"}
Oct 30 19:15:03.929635 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.929608    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" event={"ID":"0b63f080-3d3f-46ec-983c-5fe41070c214","Type":"ContainerStarted","Data":"69ff2a603f4ae3912c6286b16b8f01c423d6398fba3d62b6e4bf224737b487db"}
Oct 30 19:15:03.933020 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.932796    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" event={"ID":"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf","Type":"ContainerStarted","Data":"e517fcb189384c672a57b6f46cbd758fdacf7a7becb2110d6ddfefd9d8f8b853"}
Oct 30 19:15:03.934923 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.934895    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" event={"ID":"0e1c3881-68a5-4020-adf3-a7399af81b17","Type":"ContainerStarted","Data":"e47c3f30e09eb79425f8329daede8f1b719776b05ebee59b0eab602bd8710994"}
Oct 30 19:15:03.935177 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.935161    4267 scope.go:122] "RemoveContainer" containerID="e47c3f30e09eb79425f8329daede8f1b719776b05ebee59b0eab602bd8710994"
Oct 30 19:15:03.935553 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:03.935484    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw_odh-ai-gateway-infra(0e1c3881-68a5-4020-adf3-a7399af81b17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17"
Oct 30 19:15:03.937879 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.937852    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" event={"ID":"d8c1e641-75e1-4302-82e0-da4f6205f3d8","Type":"ContainerStarted","Data":"ced01493814b79cb2d106512d28b4a43c3e3b5c564490d8328b4553c5eef68af"}
Oct 30 19:15:03.939936 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.939911    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" event={"ID":"b98b2065-c604-4e53-b671-94573cad2963","Type":"ContainerStarted","Data":"14765256eda8a53fc6e47bb884a081187c43f885a5b99d380cdbeb8b7ea70af3"}
Oct 30 19:15:03.942222 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.942197    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" event={"ID":"48bb6f2b-8772-4579-88d8-4a347aeee899","Type":"ContainerStarted","Data":"6d651bfad87cb1024e1d3116d8ab7ab063032e594cc7ae984b5cf578101180d6"}
Oct 30 19:15:03.944386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.944367    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" event={"ID":"31bdf1c9-20f7-449c-8807-88c5fbe86b35","Type":"ContainerStarted","Data":"79f339090f2844d8a4f162e893a8bc7e24eb772c2977a8cc1be63aaae495e8f0"}
Oct 30 19:15:03.944636 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:03.944620    4267 scope.go:122] "RemoveContainer" containerID="79f339090f2844d8a4f162e893a8bc7e24eb772c2977a8cc1be63aaae495e8f0"
Oct 30 19:15:03.944829 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:03.944813    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk_odh-ai-gateway-infra(31bdf1c9-20f7-449c-8807-88c5fbe86b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35"
Oct 30 19:15:04.043716 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.043668    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" podStartSLOduration=4.043654996 podStartE2EDuration="4.043654996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:04.040239955 +0000 UTC m=+2194.928157044" watchObservedRunningTime="2026-10-30 19:15:04.043654996 +0000 UTC m=+2194.931572085"
Oct 30 19:15:04.953911 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.953634    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" podStartSLOduration=4.953614837 podStartE2EDuration="4.953614837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:04.945229324 +0000 UTC m=+2195.833146404" watchObservedRunningTime="2026-10-30 19:15:04.953614837 +0000 UTC m=+2195.841531929"
Oct 30 19:15:04.960141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.960111    4267 generic.go:356] "Generic (PLEG): container finished" podID="b98b2065-c604-4e53-b671-94573cad2963" containerID="14765256eda8a53fc6e47bb884a081187c43f885a5b99d380cdbeb8b7ea70af3" exitCode=60
Oct 30 19:15:04.960389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.960373    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" event={"ID":"b98b2065-c604-4e53-b671-94573cad2963","Type":"ContainerDied","Data":"14765256eda8a53fc6e47bb884a081187c43f885a5b99d380cdbeb8b7ea70af3"}
Oct 30 19:15:04.960593 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.960573    4267 scope.go:122] "RemoveContainer" containerID="141958baf473039129799b4c4ff9c37e1fa01c3d63f678399a2a9dc47089934c"
Oct 30 19:15:04.961341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.961320    4267 scope.go:122] "RemoveContainer" containerID="14765256eda8a53fc6e47bb884a081187c43f885a5b99d380cdbeb8b7ea70af3"
Oct 30 19:15:04.961951 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:04.961923    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc_odh-ai-gateway-infra(b98b2065-c604-4e53-b671-94573cad2963)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" podUID="b98b2065-c604-4e53-b671-94573cad2963"
Oct 30 19:15:04.967571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.967476    4267 generic.go:356] "Generic (PLEG): container finished" podID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerID="6d651bfad87cb1024e1d3116d8ab7ab063032e594cc7ae984b5cf578101180d6" exitCode=60
Oct 30 19:15:04.967720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.967693    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" event={"ID":"48bb6f2b-8772-4579-88d8-4a347aeee899","Type":"ContainerDied","Data":"6d651bfad87cb1024e1d3116d8ab7ab063032e594cc7ae984b5cf578101180d6"}
Oct 30 19:15:04.969069 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.969014    4267 scope.go:122] "RemoveContainer" containerID="6d651bfad87cb1024e1d3116d8ab7ab063032e594cc7ae984b5cf578101180d6"
Oct 30 19:15:04.969625 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:04.969484    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q_odh-ai-gateway-infra(48bb6f2b-8772-4579-88d8-4a347aeee899)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899"
Oct 30 19:15:04.970508 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.970471    4267 generic.go:356] "Generic (PLEG): container finished" podID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerID="79f339090f2844d8a4f162e893a8bc7e24eb772c2977a8cc1be63aaae495e8f0" exitCode=7
Oct 30 19:15:04.970790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.970754    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" event={"ID":"31bdf1c9-20f7-449c-8807-88c5fbe86b35","Type":"ContainerDied","Data":"79f339090f2844d8a4f162e893a8bc7e24eb772c2977a8cc1be63aaae495e8f0"}
Oct 30 19:15:04.970989 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.970972    4267 scope.go:122] "RemoveContainer" containerID="79f339090f2844d8a4f162e893a8bc7e24eb772c2977a8cc1be63aaae495e8f0"
Oct 30 19:15:04.971351 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:04.971331    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk_odh-ai-gateway-infra(31bdf1c9-20f7-449c-8807-88c5fbe86b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35"
Oct 30 19:15:04.974286 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.974263    4267 generic.go:356] "Generic (PLEG): container finished" podID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerID="f4530d361159139697fd6188be8ab0fd399738d6c4c999a39f2e4bf2a9171414" exitCode=60
Oct 30 19:15:04.975006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.974857    4267 scope.go:122] "RemoveContainer" containerID="f4530d361159139697fd6188be8ab0fd399738d6c4c999a39f2e4bf2a9171414"
Oct 30 19:15:04.975863 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.974876    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" event={"ID":"842cbd49-4cce-4500-b92b-d9b1446926c2","Type":"ContainerDied","Data":"f4530d361159139697fd6188be8ab0fd399738d6c4c999a39f2e4bf2a9171414"}
Oct 30 19:15:04.976081 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:04.975998    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5_odh-ai-gateway-infra(842cbd49-4cce-4500-b92b-d9b1446926c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2"
Oct 30 19:15:04.978860 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.978834    4267 generic.go:356] "Generic (PLEG): container finished" podID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerID="96e84f071f1eba2f1436324339c0f6cc577a128223ac5f8329cbc504e4651315" exitCode=60
Oct 30 19:15:04.978950 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.978935    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" event={"ID":"ecaf59c7-05b4-4787-8d74-fb373c68838b","Type":"ContainerDied","Data":"96e84f071f1eba2f1436324339c0f6cc577a128223ac5f8329cbc504e4651315"}
Oct 30 19:15:04.979315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.979300    4267 scope.go:122] "RemoveContainer" containerID="96e84f071f1eba2f1436324339c0f6cc577a128223ac5f8329cbc504e4651315"
Oct 30 19:15:04.979721 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:04.979702    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp_odh-ai-gateway-infra(ecaf59c7-05b4-4787-8d74-fb373c68838b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b"
Oct 30 19:15:04.985280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.985256    4267 generic.go:356] "Generic (PLEG): container finished" podID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerID="69ff2a603f4ae3912c6286b16b8f01c423d6398fba3d62b6e4bf224737b487db" exitCode=60
Oct 30 19:15:04.985388 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.985340    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" event={"ID":"0b63f080-3d3f-46ec-983c-5fe41070c214","Type":"ContainerDied","Data":"69ff2a603f4ae3912c6286b16b8f01c423d6398fba3d62b6e4bf224737b487db"}
Oct 30 19:15:04.985657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.985637    4267 scope.go:122] "RemoveContainer" containerID="69ff2a603f4ae3912c6286b16b8f01c423d6398fba3d62b6e4bf224737b487db"
Oct 30 19:15:04.985984 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:04.985958    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq_odh-ai-gateway-infra(0b63f080-3d3f-46ec-983c-5fe41070c214)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214"
Oct 30 19:15:04.988051 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.988030    4267 generic.go:356] "Generic (PLEG): container finished" podID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerID="e517fcb189384c672a57b6f46cbd758fdacf7a7becb2110d6ddfefd9d8f8b853" exitCode=60
Oct 30 19:15:04.988136 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.988112    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" event={"ID":"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf","Type":"ContainerDied","Data":"e517fcb189384c672a57b6f46cbd758fdacf7a7becb2110d6ddfefd9d8f8b853"}
Oct 30 19:15:04.988363 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.988345    4267 scope.go:122] "RemoveContainer" containerID="e517fcb189384c672a57b6f46cbd758fdacf7a7becb2110d6ddfefd9d8f8b853"
Oct 30 19:15:04.988671 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:04.988653    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7_odh-ai-gateway-infra(5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf"
Oct 30 19:15:04.988974 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.988961    4267 scope.go:122] "RemoveContainer" containerID="3d4b1b27fe54861d51cdbac7839ab3df92d858144e7ddbe14b4515a372752712"
Oct 30 19:15:04.990696 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.990673    4267 generic.go:356] "Generic (PLEG): container finished" podID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerID="e47c3f30e09eb79425f8329daede8f1b719776b05ebee59b0eab602bd8710994" exitCode=60
Oct 30 19:15:04.990792 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.990759    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" event={"ID":"0e1c3881-68a5-4020-adf3-a7399af81b17","Type":"ContainerDied","Data":"e47c3f30e09eb79425f8329daede8f1b719776b05ebee59b0eab602bd8710994"}
Oct 30 19:15:04.991154 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.991131    4267 scope.go:122] "RemoveContainer" containerID="e47c3f30e09eb79425f8329daede8f1b719776b05ebee59b0eab602bd8710994"
Oct 30 19:15:04.991681 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:04.991655    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw_odh-ai-gateway-infra(0e1c3881-68a5-4020-adf3-a7399af81b17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17"
Oct 30 19:15:04.993061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.993036    4267 generic.go:356] "Generic (PLEG): container finished" podID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerID="ced01493814b79cb2d106512d28b4a43c3e3b5c564490d8328b4553c5eef68af" exitCode=60
Oct 30 19:15:04.993200 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.993128    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" event={"ID":"d8c1e641-75e1-4302-82e0-da4f6205f3d8","Type":"ContainerDied","Data":"ced01493814b79cb2d106512d28b4a43c3e3b5c564490d8328b4553c5eef68af"}
Oct 30 19:15:04.993439 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:04.993421    4267 scope.go:122] "RemoveContainer" containerID="ced01493814b79cb2d106512d28b4a43c3e3b5c564490d8328b4553c5eef68af"
Oct 30 19:15:04.993780 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:04.993759    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z_odh-ai-gateway-infra(d8c1e641-75e1-4302-82e0-da4f6205f3d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8"
Oct 30 19:15:05.006728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:05.006705    4267 scope.go:122] "RemoveContainer" containerID="a216d0c9c3a4dffdd0e331e3d9dd8efe7b58d8f24627c5e8c41fe4fe0e69387b"
Oct 30 19:15:05.027137 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:05.027105    4267 scope.go:122] "RemoveContainer" containerID="3bd538c19ddd51078a165893de01b682b62b12cad78f997099c7e90d6e092608"
Oct 30 19:15:05.049745 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:05.049718    4267 scope.go:122] "RemoveContainer" containerID="de0bea6141cb33b62e7856755bf159914a54ed1540c2e2ef9f9dfda4488d1cac"
Oct 30 19:15:05.065564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:05.065483    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" podStartSLOduration=5.065463364 podStartE2EDuration="5.065463364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:05.058974386 +0000 UTC m=+2195.946891481" watchObservedRunningTime="2026-10-30 19:15:05.065463364 +0000 UTC m=+2195.953380454"
Oct 30 19:15:05.078560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:05.078539    4267 scope.go:122] "RemoveContainer" containerID="32d4efbea1b441435093a92f3d452137d0fd432262780ed577fdc03e00907a43"
Oct 30 19:15:05.095858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:05.095829    4267 scope.go:122] "RemoveContainer" containerID="20533b65057d48cf613e3cef19ef30dcb14ba3a5a239be1ca27c82b6ab262807"
Oct 30 19:15:05.113390 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:05.113367    4267 scope.go:122] "RemoveContainer" containerID="885083ee0064d6c09f00f70e451c87bdd41aaf8ee9744fd7c544d24b589e2c30"
Oct 30 19:15:05.128996 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:05.128965    4267 scope.go:122] "RemoveContainer" containerID="1b04dbcbc4639b0cdda6e95e2e986ebebb262927b9841c9d5ea12f26a326006f"
Oct 30 19:15:06.019514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:06.019336    4267 scope.go:122] "RemoveContainer" containerID="ced01493814b79cb2d106512d28b4a43c3e3b5c564490d8328b4553c5eef68af"
Oct 30 19:15:06.019988 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:06.019875    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z_odh-ai-gateway-infra(d8c1e641-75e1-4302-82e0-da4f6205f3d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8"
Oct 30 19:15:06.030398 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:06.030357    4267 scope.go:122] "RemoveContainer" containerID="14765256eda8a53fc6e47bb884a081187c43f885a5b99d380cdbeb8b7ea70af3"
Oct 30 19:15:06.030802 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:06.030768    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc_odh-ai-gateway-infra(b98b2065-c604-4e53-b671-94573cad2963)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" podUID="b98b2065-c604-4e53-b671-94573cad2963"
Oct 30 19:15:06.066512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:06.066456    4267 scope.go:122] "RemoveContainer" containerID="6d651bfad87cb1024e1d3116d8ab7ab063032e594cc7ae984b5cf578101180d6"
Oct 30 19:15:06.067405 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:06.067373    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q_odh-ai-gateway-infra(48bb6f2b-8772-4579-88d8-4a347aeee899)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899"
Oct 30 19:15:06.072779 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:06.072715    4267 scope.go:122] "RemoveContainer" containerID="79f339090f2844d8a4f162e893a8bc7e24eb772c2977a8cc1be63aaae495e8f0"
Oct 30 19:15:06.073401 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:06.073372    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk_odh-ai-gateway-infra(31bdf1c9-20f7-449c-8807-88c5fbe86b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35"
Oct 30 19:15:06.076475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:06.076435    4267 scope.go:122] "RemoveContainer" containerID="f4530d361159139697fd6188be8ab0fd399738d6c4c999a39f2e4bf2a9171414"
Oct 30 19:15:06.077061 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:06.077032    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5_odh-ai-gateway-infra(842cbd49-4cce-4500-b92b-d9b1446926c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2"
Oct 30 19:15:06.081487 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:06.081447    4267 scope.go:122] "RemoveContainer" containerID="96e84f071f1eba2f1436324339c0f6cc577a128223ac5f8329cbc504e4651315"
Oct 30 19:15:06.094723 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:06.094684    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp_odh-ai-gateway-infra(ecaf59c7-05b4-4787-8d74-fb373c68838b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b"
Oct 30 19:15:06.094978 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:06.094912    4267 scope.go:122] "RemoveContainer" containerID="69ff2a603f4ae3912c6286b16b8f01c423d6398fba3d62b6e4bf224737b487db"
Oct 30 19:15:06.096886 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:06.096857    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq_odh-ai-gateway-infra(0b63f080-3d3f-46ec-983c-5fe41070c214)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214"
Oct 30 19:15:06.098328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:06.097010    4267 scope.go:122] "RemoveContainer" containerID="e517fcb189384c672a57b6f46cbd758fdacf7a7becb2110d6ddfefd9d8f8b853"
Oct 30 19:15:06.098737 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:06.098712    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7_odh-ai-gateway-infra(5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf"
Oct 30 19:15:06.098880 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:06.098839    4267 scope.go:122] "RemoveContainer" containerID="e47c3f30e09eb79425f8329daede8f1b719776b05ebee59b0eab602bd8710994"
Oct 30 19:15:06.099201 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:06.099180    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw_odh-ai-gateway-infra(0e1c3881-68a5-4020-adf3-a7399af81b17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17"
Oct 30 19:15:12.298819 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:12.298726    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"]
Oct 30 19:15:12.363136 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:12.363092    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"]
Oct 30 19:15:15.213945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:15.213908    4267 scope.go:122] "RemoveContainer" containerID="96e84f071f1eba2f1436324339c0f6cc577a128223ac5f8329cbc504e4651315"
Oct 30 19:15:15.214457 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:15.214433    4267 scope.go:122] "RemoveContainer" containerID="6d651bfad87cb1024e1d3116d8ab7ab063032e594cc7ae984b5cf578101180d6"
Oct 30 19:15:15.215039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:15.215016    4267 scope.go:122] "RemoveContainer" containerID="79f339090f2844d8a4f162e893a8bc7e24eb772c2977a8cc1be63aaae495e8f0"
Oct 30 19:15:16.213480 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.213442    4267 scope.go:122] "RemoveContainer" containerID="ced01493814b79cb2d106512d28b4a43c3e3b5c564490d8328b4553c5eef68af"
Oct 30 19:15:16.213698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.213666    4267 scope.go:122] "RemoveContainer" containerID="69ff2a603f4ae3912c6286b16b8f01c423d6398fba3d62b6e4bf224737b487db"
Oct 30 19:15:16.252132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.252094    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" event={"ID":"48bb6f2b-8772-4579-88d8-4a347aeee899","Type":"ContainerStarted","Data":"57587c2fbfb5cabf0b7027e51bf40437ea4bf0cd66fd99f5ddb9962fd37a6231"}
Oct 30 19:15:16.254034 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.254000    4267 generic.go:356] "Generic (PLEG): container finished" podID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerID="5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d" exitCode=0
Oct 30 19:15:16.254153 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.254073    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" event={"ID":"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5","Type":"ContainerDied","Data":"5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d"}
Oct 30 19:15:16.254621 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.254603    4267 scope.go:122] "RemoveContainer" containerID="5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d"
Oct 30 19:15:16.256680 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.256653    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" event={"ID":"31bdf1c9-20f7-449c-8807-88c5fbe86b35","Type":"ContainerStarted","Data":"f18dbc683f35b522f7febdb5cc3b1dbefd50150e996213a8f3121689e38ad22a"}
Oct 30 19:15:16.256825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.256670    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:16.256977 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.256960    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:16.257093 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.257077    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:16.260598 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:16.260519    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" event={"ID":"ecaf59c7-05b4-4787-8d74-fb373c68838b","Type":"ContainerStarted","Data":"0334cfe536ba34a9709b8efc5fa3a3b238dfe1ef160de150fff7c99460ab138d"}
Oct 30 19:15:17.214049 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.214012    4267 scope.go:122] "RemoveContainer" containerID="f4530d361159139697fd6188be8ab0fd399738d6c4c999a39f2e4bf2a9171414"
Oct 30 19:15:17.214462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.214438    4267 scope.go:122] "RemoveContainer" containerID="14765256eda8a53fc6e47bb884a081187c43f885a5b99d380cdbeb8b7ea70af3"
Oct 30 19:15:17.214855 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.214792    4267 scope.go:122] "RemoveContainer" containerID="e517fcb189384c672a57b6f46cbd758fdacf7a7becb2110d6ddfefd9d8f8b853"
Oct 30 19:15:17.272747 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.272717    4267 generic.go:356] "Generic (PLEG): container finished" podID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerID="57587c2fbfb5cabf0b7027e51bf40437ea4bf0cd66fd99f5ddb9962fd37a6231" exitCode=60
Oct 30 19:15:17.273204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.272785    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" event={"ID":"48bb6f2b-8772-4579-88d8-4a347aeee899","Type":"ContainerDied","Data":"57587c2fbfb5cabf0b7027e51bf40437ea4bf0cd66fd99f5ddb9962fd37a6231"}
Oct 30 19:15:17.273204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.272842    4267 scope.go:122] "RemoveContainer" containerID="6d651bfad87cb1024e1d3116d8ab7ab063032e594cc7ae984b5cf578101180d6"
Oct 30 19:15:17.273362 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.273345    4267 scope.go:122] "RemoveContainer" containerID="57587c2fbfb5cabf0b7027e51bf40437ea4bf0cd66fd99f5ddb9962fd37a6231"
Oct 30 19:15:17.273741 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:17.273719    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q_odh-ai-gateway-infra(48bb6f2b-8772-4579-88d8-4a347aeee899)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899"
Oct 30 19:15:17.275698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.275678    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" event={"ID":"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5","Type":"ContainerStarted","Data":"877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777"}
Oct 30 19:15:17.278001 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.277973    4267 generic.go:356] "Generic (PLEG): container finished" podID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerID="f18dbc683f35b522f7febdb5cc3b1dbefd50150e996213a8f3121689e38ad22a" exitCode=60
Oct 30 19:15:17.278106 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.278021    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" event={"ID":"31bdf1c9-20f7-449c-8807-88c5fbe86b35","Type":"ContainerDied","Data":"f18dbc683f35b522f7febdb5cc3b1dbefd50150e996213a8f3121689e38ad22a"}
Oct 30 19:15:17.278376 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.278360    4267 scope.go:122] "RemoveContainer" containerID="f18dbc683f35b522f7febdb5cc3b1dbefd50150e996213a8f3121689e38ad22a"
Oct 30 19:15:17.278715 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:17.278691    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk_odh-ai-gateway-infra(31bdf1c9-20f7-449c-8807-88c5fbe86b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35"
Oct 30 19:15:17.280911 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.280266    4267 generic.go:356] "Generic (PLEG): container finished" podID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerID="0334cfe536ba34a9709b8efc5fa3a3b238dfe1ef160de150fff7c99460ab138d" exitCode=60
Oct 30 19:15:17.280911 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.280329    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" event={"ID":"ecaf59c7-05b4-4787-8d74-fb373c68838b","Type":"ContainerDied","Data":"0334cfe536ba34a9709b8efc5fa3a3b238dfe1ef160de150fff7c99460ab138d"}
Oct 30 19:15:17.280911 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.280631    4267 scope.go:122] "RemoveContainer" containerID="0334cfe536ba34a9709b8efc5fa3a3b238dfe1ef160de150fff7c99460ab138d"
Oct 30 19:15:17.280911 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:17.280874    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp_odh-ai-gateway-infra(ecaf59c7-05b4-4787-8d74-fb373c68838b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b"
Oct 30 19:15:17.282465 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.282444    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" event={"ID":"0b63f080-3d3f-46ec-983c-5fe41070c214","Type":"ContainerStarted","Data":"18cd9227afbecff67ed584234c16287bd6e63b3ace3a88cae0831389dcd63ce7"}
Oct 30 19:15:17.284373 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.284351    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" event={"ID":"d8c1e641-75e1-4302-82e0-da4f6205f3d8","Type":"ContainerStarted","Data":"5ed4fb0665795bc3db3fbd4f21a9cc5189e9b0b5497e41e7811d8947b0965acb"}
Oct 30 19:15:17.296993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.296974    4267 scope.go:122] "RemoveContainer" containerID="79f339090f2844d8a4f162e893a8bc7e24eb772c2977a8cc1be63aaae495e8f0"
Oct 30 19:15:17.317156 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.317109    4267 scope.go:122] "RemoveContainer" containerID="96e84f071f1eba2f1436324339c0f6cc577a128223ac5f8329cbc504e4651315"
Oct 30 19:15:17.380750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.380703    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"]
Oct 30 19:15:17.497380 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.497299    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"]
Oct 30 19:15:17.606968 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.606930    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"]
Oct 30 19:15:17.832571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.830875    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:15:17.856512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.842638    4267 patch_prober.go:29] interesting pod/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:15:17.856512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:17.842707    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:15:18.225519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.219906    4267 scope.go:122] "RemoveContainer" containerID="e47c3f30e09eb79425f8329daede8f1b719776b05ebee59b0eab602bd8710994"
Oct 30 19:15:18.252124 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.252096    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:15:18.295907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.295880    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" event={"ID":"842cbd49-4cce-4500-b92b-d9b1446926c2","Type":"ContainerStarted","Data":"b8e0fe6308ec7468e7c55aed85eb371ef3e8d509a2dee093fd87e0c37f251545"}
Oct 30 19:15:18.299555 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.299529    4267 generic.go:356] "Generic (PLEG): container finished" podID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerID="18cd9227afbecff67ed584234c16287bd6e63b3ace3a88cae0831389dcd63ce7" exitCode=60
Oct 30 19:15:18.299673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.299531    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" event={"ID":"0b63f080-3d3f-46ec-983c-5fe41070c214","Type":"ContainerDied","Data":"18cd9227afbecff67ed584234c16287bd6e63b3ace3a88cae0831389dcd63ce7"}
Oct 30 19:15:18.299673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.299655    4267 scope.go:122] "RemoveContainer" containerID="69ff2a603f4ae3912c6286b16b8f01c423d6398fba3d62b6e4bf224737b487db"
Oct 30 19:15:18.300394 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.300372    4267 scope.go:122] "RemoveContainer" containerID="18cd9227afbecff67ed584234c16287bd6e63b3ace3a88cae0831389dcd63ce7"
Oct 30 19:15:18.300805 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:18.300775    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq_odh-ai-gateway-infra(0b63f080-3d3f-46ec-983c-5fe41070c214)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214"
Oct 30 19:15:18.302000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.301887    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" event={"ID":"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf","Type":"ContainerStarted","Data":"c4fa1379a205db380358593e476f3dc2403fb8d6f55ef90109f1a3ca8b45e659"}
Oct 30 19:15:18.304300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.304276    4267 generic.go:356] "Generic (PLEG): container finished" podID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerID="5ed4fb0665795bc3db3fbd4f21a9cc5189e9b0b5497e41e7811d8947b0965acb" exitCode=60
Oct 30 19:15:18.304532 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.304513    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" event={"ID":"d8c1e641-75e1-4302-82e0-da4f6205f3d8","Type":"ContainerDied","Data":"5ed4fb0665795bc3db3fbd4f21a9cc5189e9b0b5497e41e7811d8947b0965acb"}
Oct 30 19:15:18.304866 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.304849    4267 scope.go:122] "RemoveContainer" containerID="5ed4fb0665795bc3db3fbd4f21a9cc5189e9b0b5497e41e7811d8947b0965acb"
Oct 30 19:15:18.305238 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:18.305207    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z_odh-ai-gateway-infra(d8c1e641-75e1-4302-82e0-da4f6205f3d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8"
Oct 30 19:15:18.308233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.308211    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" event={"ID":"b98b2065-c604-4e53-b671-94573cad2963","Type":"ContainerStarted","Data":"a069b89f74e9105999c0baf6180442d0b843d0291290c19892190175aab01ba7"}
Oct 30 19:15:18.358130 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.356953    4267 scope.go:122] "RemoveContainer" containerID="ced01493814b79cb2d106512d28b4a43c3e3b5c564490d8328b4553c5eef68af"
Oct 30 19:15:18.659924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.659815    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"]
Oct 30 19:15:18.797813 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.797786    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"
Oct 30 19:15:18.801796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.801772    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"
Oct 30 19:15:18.851530 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.846681    4267 patch_prober.go:29] interesting pod/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:15:18.851530 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.847766    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:15:18.900656 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.900629    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"
Oct 30 19:15:18.973649 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.973608    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecaf59c7-05b4-4787-8d74-fb373c68838b-kube-api-access-76c6s\" (UniqueName: \"kubernetes.io/projected/ecaf59c7-05b4-4787-8d74-fb373c68838b-kube-api-access-76c6s\") pod \"ecaf59c7-05b4-4787-8d74-fb373c68838b\" (UID: \"ecaf59c7-05b4-4787-8d74-fb373c68838b\") "
Oct 30 19:15:18.973816 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.973687    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31bdf1c9-20f7-449c-8807-88c5fbe86b35-kube-api-access-mq7lk\" (UniqueName: \"kubernetes.io/projected/31bdf1c9-20f7-449c-8807-88c5fbe86b35-kube-api-access-mq7lk\") pod \"31bdf1c9-20f7-449c-8807-88c5fbe86b35\" (UID: \"31bdf1c9-20f7-449c-8807-88c5fbe86b35\") "
Oct 30 19:15:18.973816 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.973789    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31bdf1c9-20f7-449c-8807-88c5fbe86b35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bdf1c9-20f7-449c-8807-88c5fbe86b35-maas-api-service-ca\") pod \"31bdf1c9-20f7-449c-8807-88c5fbe86b35\" (UID: \"31bdf1c9-20f7-449c-8807-88c5fbe86b35\") "
Oct 30 19:15:18.973983 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.973968    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecaf59c7-05b4-4787-8d74-fb373c68838b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecaf59c7-05b4-4787-8d74-fb373c68838b-maas-api-service-ca\") pod \"ecaf59c7-05b4-4787-8d74-fb373c68838b\" (UID: \"ecaf59c7-05b4-4787-8d74-fb373c68838b\") "
Oct 30 19:15:18.974359 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.974332    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31bdf1c9-20f7-449c-8807-88c5fbe86b35-maas-api-service-ca" pod "31bdf1c9-20f7-449c-8807-88c5fbe86b35" (UID: "31bdf1c9-20f7-449c-8807-88c5fbe86b35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:18.974827 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.974796    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecaf59c7-05b4-4787-8d74-fb373c68838b-maas-api-service-ca" pod "ecaf59c7-05b4-4787-8d74-fb373c68838b" (UID: "ecaf59c7-05b4-4787-8d74-fb373c68838b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:18.974942 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.974898    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31bdf1c9-20f7-449c-8807-88c5fbe86b35-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.975943 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.975922    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bdf1c9-20f7-449c-8807-88c5fbe86b35-kube-api-access-mq7lk" pod "31bdf1c9-20f7-449c-8807-88c5fbe86b35" (UID: "31bdf1c9-20f7-449c-8807-88c5fbe86b35"). InnerVolumeSpecName "kube-api-access-mq7lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:18.976977 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.976959    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecaf59c7-05b4-4787-8d74-fb373c68838b-kube-api-access-76c6s" pod "ecaf59c7-05b4-4787-8d74-fb373c68838b" (UID: "ecaf59c7-05b4-4787-8d74-fb373c68838b"). InnerVolumeSpecName "kube-api-access-76c6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:18.998811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:18.998748    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"]
Oct 30 19:15:19.075890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.075859    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bb6f2b-8772-4579-88d8-4a347aeee899-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bb6f2b-8772-4579-88d8-4a347aeee899-maas-api-service-ca\") pod \"48bb6f2b-8772-4579-88d8-4a347aeee899\" (UID: \"48bb6f2b-8772-4579-88d8-4a347aeee899\") "
Oct 30 19:15:19.076068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.076011    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bb6f2b-8772-4579-88d8-4a347aeee899-kube-api-access-99rzn\" (UniqueName: \"kubernetes.io/projected/48bb6f2b-8772-4579-88d8-4a347aeee899-kube-api-access-99rzn\") pod \"48bb6f2b-8772-4579-88d8-4a347aeee899\" (UID: \"48bb6f2b-8772-4579-88d8-4a347aeee899\") "
Oct 30 19:15:19.076281 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.076258    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bb6f2b-8772-4579-88d8-4a347aeee899-maas-api-service-ca" pod "48bb6f2b-8772-4579-88d8-4a347aeee899" (UID: "48bb6f2b-8772-4579-88d8-4a347aeee899"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:19.076670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.076655    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecaf59c7-05b4-4787-8d74-fb373c68838b-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.076730 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.076673    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76c6s\" (UniqueName: \"kubernetes.io/projected/ecaf59c7-05b4-4787-8d74-fb373c68838b-kube-api-access-76c6s\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.076730 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.076684    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7lk\" (UniqueName: \"kubernetes.io/projected/31bdf1c9-20f7-449c-8807-88c5fbe86b35-kube-api-access-mq7lk\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.076730 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.076693    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bb6f2b-8772-4579-88d8-4a347aeee899-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.078123 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.078100    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bb6f2b-8772-4579-88d8-4a347aeee899-kube-api-access-99rzn" pod "48bb6f2b-8772-4579-88d8-4a347aeee899" (UID: "48bb6f2b-8772-4579-88d8-4a347aeee899"). InnerVolumeSpecName "kube-api-access-99rzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:19.178520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.178451    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99rzn\" (UniqueName: \"kubernetes.io/projected/48bb6f2b-8772-4579-88d8-4a347aeee899-kube-api-access-99rzn\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.367012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.366972    4267 generic.go:356] "Generic (PLEG): container finished" podID="b98b2065-c604-4e53-b671-94573cad2963" containerID="a069b89f74e9105999c0baf6180442d0b843d0291290c19892190175aab01ba7" exitCode=60
Oct 30 19:15:19.367535 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.367070    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" event={"ID":"b98b2065-c604-4e53-b671-94573cad2963","Type":"ContainerDied","Data":"a069b89f74e9105999c0baf6180442d0b843d0291290c19892190175aab01ba7"}
Oct 30 19:15:19.367535 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.367129    4267 scope.go:122] "RemoveContainer" containerID="14765256eda8a53fc6e47bb884a081187c43f885a5b99d380cdbeb8b7ea70af3"
Oct 30 19:15:19.367651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.367633    4267 scope.go:122] "RemoveContainer" containerID="a069b89f74e9105999c0baf6180442d0b843d0291290c19892190175aab01ba7"
Oct 30 19:15:19.368010 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:19.367985    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc_odh-ai-gateway-infra(b98b2065-c604-4e53-b671-94573cad2963)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" podUID="b98b2065-c604-4e53-b671-94573cad2963"
Oct 30 19:15:19.371194 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.371174    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"
Oct 30 19:15:19.371305 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.371206    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q" event={"ID":"48bb6f2b-8772-4579-88d8-4a347aeee899","Type":"ContainerDied","Data":"b8af50729e204e3228736e28d953e81beeb8c6c35de40913151f92ae452a5911"}
Oct 30 19:15:19.373415 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.373402    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"
Oct 30 19:15:19.375747 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.373544    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk" event={"ID":"31bdf1c9-20f7-449c-8807-88c5fbe86b35","Type":"ContainerDied","Data":"c0474b33b337e5b014a62b3e1005d4ea1aec5af9ab4fc5b7403359a31fd3eb74"}
Oct 30 19:15:19.375859 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.375783    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" event={"ID":"842cbd49-4cce-4500-b92b-d9b1446926c2","Type":"ContainerDied","Data":"b8e0fe6308ec7468e7c55aed85eb371ef3e8d509a2dee093fd87e0c37f251545"}
Oct 30 19:15:19.376259 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.375522    4267 generic.go:356] "Generic (PLEG): container finished" podID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerID="b8e0fe6308ec7468e7c55aed85eb371ef3e8d509a2dee093fd87e0c37f251545" exitCode=60
Oct 30 19:15:19.376259 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.376252    4267 scope.go:122] "RemoveContainer" containerID="b8e0fe6308ec7468e7c55aed85eb371ef3e8d509a2dee093fd87e0c37f251545"
Oct 30 19:15:19.376595 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:19.376567    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5_odh-ai-gateway-infra(842cbd49-4cce-4500-b92b-d9b1446926c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2"
Oct 30 19:15:19.377691 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.377611    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp" event={"ID":"ecaf59c7-05b4-4787-8d74-fb373c68838b","Type":"ContainerDied","Data":"8b9fb44d70e68395099c40d09a2c1a0e9f7761233242fcc470172467fb999e4a"}
Oct 30 19:15:19.377792 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.377690    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"
Oct 30 19:15:19.388747 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.388662    4267 generic.go:356] "Generic (PLEG): container finished" podID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerID="c4fa1379a205db380358593e476f3dc2403fb8d6f55ef90109f1a3ca8b45e659" exitCode=60
Oct 30 19:15:19.388862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.388764    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" event={"ID":"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf","Type":"ContainerDied","Data":"c4fa1379a205db380358593e476f3dc2403fb8d6f55ef90109f1a3ca8b45e659"}
Oct 30 19:15:19.389180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.389127    4267 scope.go:122] "RemoveContainer" containerID="c4fa1379a205db380358593e476f3dc2403fb8d6f55ef90109f1a3ca8b45e659"
Oct 30 19:15:19.389523 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:19.389433    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7_odh-ai-gateway-infra(5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf"
Oct 30 19:15:19.391965 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.391936    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" event={"ID":"0e1c3881-68a5-4020-adf3-a7399af81b17","Type":"ContainerStarted","Data":"7b89d5a9c5f459fae5589d6d57b673e3ccb535b5ac7f83068de284a69af6efdf"}
Oct 30 19:15:19.395060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.395044    4267 scope.go:122] "RemoveContainer" containerID="57587c2fbfb5cabf0b7027e51bf40437ea4bf0cd66fd99f5ddb9962fd37a6231"
Oct 30 19:15:19.406356 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.406318    4267 scope.go:122] "RemoveContainer" containerID="f18dbc683f35b522f7febdb5cc3b1dbefd50150e996213a8f3121689e38ad22a"
Oct 30 19:15:19.599652 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.592649    4267 scope.go:122] "RemoveContainer" containerID="f4530d361159139697fd6188be8ab0fd399738d6c4c999a39f2e4bf2a9171414"
Oct 30 19:15:19.700836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.700809    4267 scope.go:122] "RemoveContainer" containerID="0334cfe536ba34a9709b8efc5fa3a3b238dfe1ef160de150fff7c99460ab138d"
Oct 30 19:15:19.764094 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.763977    4267 scope.go:122] "RemoveContainer" containerID="e517fcb189384c672a57b6f46cbd758fdacf7a7becb2110d6ddfefd9d8f8b853"
Oct 30 19:15:19.842885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.842796    4267 patch_prober.go:29] interesting pod/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:15:19.843129 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.843098    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:15:19.953386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.953315    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"]
Oct 30 19:15:19.965597 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:19.960577    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-611053-29889795-m5tlp"]
Oct 30 19:15:20.057539 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.053522    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"]
Oct 30 19:15:20.057743 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.057723    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"
Oct 30 19:15:20.060132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.060110    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"
Oct 30 19:15:20.187403 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.187355    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"]
Oct 30 19:15:20.188187 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.188150    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b63f080-3d3f-46ec-983c-5fe41070c214-kube-api-access-r4qg2\" (UniqueName: \"kubernetes.io/projected/0b63f080-3d3f-46ec-983c-5fe41070c214-kube-api-access-r4qg2\") pod \"0b63f080-3d3f-46ec-983c-5fe41070c214\" (UID: \"0b63f080-3d3f-46ec-983c-5fe41070c214\") "
Oct 30 19:15:20.188545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.188439    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c1e641-75e1-4302-82e0-da4f6205f3d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c1e641-75e1-4302-82e0-da4f6205f3d8-maas-api-service-ca\") pod \"d8c1e641-75e1-4302-82e0-da4f6205f3d8\" (UID: \"d8c1e641-75e1-4302-82e0-da4f6205f3d8\") "
Oct 30 19:15:20.188796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.188763    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c1e641-75e1-4302-82e0-da4f6205f3d8-kube-api-access-wd8lr\" (UniqueName: \"kubernetes.io/projected/d8c1e641-75e1-4302-82e0-da4f6205f3d8-kube-api-access-wd8lr\") pod \"d8c1e641-75e1-4302-82e0-da4f6205f3d8\" (UID: \"d8c1e641-75e1-4302-82e0-da4f6205f3d8\") "
Oct 30 19:15:20.188910 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.188809    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b63f080-3d3f-46ec-983c-5fe41070c214-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b63f080-3d3f-46ec-983c-5fe41070c214-maas-api-service-ca\") pod \"0b63f080-3d3f-46ec-983c-5fe41070c214\" (UID: \"0b63f080-3d3f-46ec-983c-5fe41070c214\") "
Oct 30 19:15:20.189353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.189331    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c1e641-75e1-4302-82e0-da4f6205f3d8-maas-api-service-ca" pod "d8c1e641-75e1-4302-82e0-da4f6205f3d8" (UID: "d8c1e641-75e1-4302-82e0-da4f6205f3d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.189889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.189866    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c1e641-75e1-4302-82e0-da4f6205f3d8-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.190853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.189906    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b63f080-3d3f-46ec-983c-5fe41070c214-maas-api-service-ca" pod "0b63f080-3d3f-46ec-983c-5fe41070c214" (UID: "0b63f080-3d3f-46ec-983c-5fe41070c214"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.193128 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.193087    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b63f080-3d3f-46ec-983c-5fe41070c214-kube-api-access-r4qg2" pod "0b63f080-3d3f-46ec-983c-5fe41070c214" (UID: "0b63f080-3d3f-46ec-983c-5fe41070c214"). InnerVolumeSpecName "kube-api-access-r4qg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.195374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.195351    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c1e641-75e1-4302-82e0-da4f6205f3d8-kube-api-access-wd8lr" pod "d8c1e641-75e1-4302-82e0-da4f6205f3d8" (UID: "d8c1e641-75e1-4302-82e0-da4f6205f3d8"). InnerVolumeSpecName "kube-api-access-wd8lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.203460 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:20.203418    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:15:20.206631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.206613    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"]
Oct 30 19:15:20.250185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.250147    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e0fc1f-29889795-skflk"]
Oct 30 19:15:20.292165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.292118    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd8lr\" (UniqueName: \"kubernetes.io/projected/d8c1e641-75e1-4302-82e0-da4f6205f3d8-kube-api-access-wd8lr\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.292165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.292161    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b63f080-3d3f-46ec-983c-5fe41070c214-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.292165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.292172    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4qg2\" (UniqueName: \"kubernetes.io/projected/0b63f080-3d3f-46ec-983c-5fe41070c214-kube-api-access-r4qg2\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.351556 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.351516    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"]
Oct 30 19:15:20.358467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.358434    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8da6e2-29889795-nbw4q"]
Oct 30 19:15:20.362592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.362565    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"]
Oct 30 19:15:20.404652 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.404626    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"
Oct 30 19:15:20.405158 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.404626    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq" event={"ID":"0b63f080-3d3f-46ec-983c-5fe41070c214","Type":"ContainerDied","Data":"56c35c742c907eccc475381d5c81b8a9065ad0ebee70d05bf1baa82bd9772145"}
Oct 30 19:15:20.405158 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.404779    4267 scope.go:122] "RemoveContainer" containerID="18cd9227afbecff67ed584234c16287bd6e63b3ace3a88cae0831389dcd63ce7"
Oct 30 19:15:20.428727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.427215    4267 generic.go:356] "Generic (PLEG): container finished" podID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerID="7b89d5a9c5f459fae5589d6d57b673e3ccb535b5ac7f83068de284a69af6efdf" exitCode=60
Oct 30 19:15:20.428727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.427364    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" event={"ID":"0e1c3881-68a5-4020-adf3-a7399af81b17","Type":"ContainerDied","Data":"7b89d5a9c5f459fae5589d6d57b673e3ccb535b5ac7f83068de284a69af6efdf"}
Oct 30 19:15:20.428727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.427899    4267 scope.go:122] "RemoveContainer" containerID="7b89d5a9c5f459fae5589d6d57b673e3ccb535b5ac7f83068de284a69af6efdf"
Oct 30 19:15:20.428727 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:20.428306    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw_odh-ai-gateway-infra(0e1c3881-68a5-4020-adf3-a7399af81b17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17"
Oct 30 19:15:20.457553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.457527    4267 scope.go:122] "RemoveContainer" containerID="e47c3f30e09eb79425f8329daede8f1b719776b05ebee59b0eab602bd8710994"
Oct 30 19:15:20.457907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.457820    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z" event={"ID":"d8c1e641-75e1-4302-82e0-da4f6205f3d8","Type":"ContainerDied","Data":"ffc2fb8953a2c0746c6494af448df0e82f2e9ac3a03e9bda0ea0315af529f98d"}
Oct 30 19:15:20.458083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.458069    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"
Oct 30 19:15:20.557520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.556014    4267 scope.go:122] "RemoveContainer" containerID="5ed4fb0665795bc3db3fbd4f21a9cc5189e9b0b5497e41e7811d8947b0965acb"
Oct 30 19:15:20.834869 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.834607    4267 patch_prober.go:29] interesting pod/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:15:20.834869 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:20.834664    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:15:21.204429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.204113    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"
Oct 30 19:15:21.274852 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.274820    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" path="/var/lib/kubelet/pods/31bdf1c9-20f7-449c-8807-88c5fbe86b35/volumes"
Oct 30 19:15:21.275768 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.275744    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899" path="/var/lib/kubelet/pods/48bb6f2b-8772-4579-88d8-4a347aeee899/volumes"
Oct 30 19:15:21.278218 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.277957    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b" path="/var/lib/kubelet/pods/ecaf59c7-05b4-4787-8d74-fb373c68838b/volumes"
Oct 30 19:15:21.284638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.281317    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-maas-api-service-ca\") pod \"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf\" (UID: \"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf\") "
Oct 30 19:15:21.284638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.281479    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-kube-api-access-cshtb\" (UniqueName: \"kubernetes.io/projected/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-kube-api-access-cshtb\") pod \"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf\" (UID: \"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf\") "
Oct 30 19:15:21.284638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.281960    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-maas-api-service-ca" pod "5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" (UID: "5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.284638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.282266    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.285396 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.285367    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-kube-api-access-cshtb" pod "5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" (UID: "5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf"). InnerVolumeSpecName "kube-api-access-cshtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.361639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.361616    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"
Oct 30 19:15:21.365998 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.365974    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"
Oct 30 19:15:21.383524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.383482    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98b2065-c604-4e53-b671-94573cad2963-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98b2065-c604-4e53-b671-94573cad2963-maas-api-service-ca\") pod \"b98b2065-c604-4e53-b671-94573cad2963\" (UID: \"b98b2065-c604-4e53-b671-94573cad2963\") "
Oct 30 19:15:21.383628 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.383558    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98b2065-c604-4e53-b671-94573cad2963-kube-api-access-fxstk\" (UniqueName: \"kubernetes.io/projected/b98b2065-c604-4e53-b671-94573cad2963-kube-api-access-fxstk\") pod \"b98b2065-c604-4e53-b671-94573cad2963\" (UID: \"b98b2065-c604-4e53-b671-94573cad2963\") "
Oct 30 19:15:21.383913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.383887    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98b2065-c604-4e53-b671-94573cad2963-maas-api-service-ca" pod "b98b2065-c604-4e53-b671-94573cad2963" (UID: "b98b2065-c604-4e53-b671-94573cad2963"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.383956 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.383940    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cshtb\" (UniqueName: \"kubernetes.io/projected/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf-kube-api-access-cshtb\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.385758 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.385736    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98b2065-c604-4e53-b671-94573cad2963-kube-api-access-fxstk" pod "b98b2065-c604-4e53-b671-94573cad2963" (UID: "b98b2065-c604-4e53-b671-94573cad2963"). InnerVolumeSpecName "kube-api-access-fxstk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.484534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.484461    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842cbd49-4cce-4500-b92b-d9b1446926c2-kube-api-access-q6dtb\" (UniqueName: \"kubernetes.io/projected/842cbd49-4cce-4500-b92b-d9b1446926c2-kube-api-access-q6dtb\") pod \"842cbd49-4cce-4500-b92b-d9b1446926c2\" (UID: \"842cbd49-4cce-4500-b92b-d9b1446926c2\") "
Oct 30 19:15:21.485005 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.484761    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842cbd49-4cce-4500-b92b-d9b1446926c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842cbd49-4cce-4500-b92b-d9b1446926c2-maas-api-service-ca\") pod \"842cbd49-4cce-4500-b92b-d9b1446926c2\" (UID: \"842cbd49-4cce-4500-b92b-d9b1446926c2\") "
Oct 30 19:15:21.485089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.485058    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842cbd49-4cce-4500-b92b-d9b1446926c2-maas-api-service-ca" pod "842cbd49-4cce-4500-b92b-d9b1446926c2" (UID: "842cbd49-4cce-4500-b92b-d9b1446926c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.485348 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.485329    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98b2065-c604-4e53-b671-94573cad2963-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.485348 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.485347    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxstk\" (UniqueName: \"kubernetes.io/projected/b98b2065-c604-4e53-b671-94573cad2963-kube-api-access-fxstk\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.485542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.485358    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842cbd49-4cce-4500-b92b-d9b1446926c2-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.485625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.485603    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"
Oct 30 19:15:21.485675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.485647    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7" event={"ID":"5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf","Type":"ContainerDied","Data":"a41400e0d6243559f241f9bb79b0e25fe7a83eff8ffdf01f0e8300b213da6790"}
Oct 30 19:15:21.486058 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.485749    4267 scope.go:122] "RemoveContainer" containerID="c4fa1379a205db380358593e476f3dc2403fb8d6f55ef90109f1a3ca8b45e659"
Oct 30 19:15:21.487252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.486779    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842cbd49-4cce-4500-b92b-d9b1446926c2-kube-api-access-q6dtb" pod "842cbd49-4cce-4500-b92b-d9b1446926c2" (UID: "842cbd49-4cce-4500-b92b-d9b1446926c2"). InnerVolumeSpecName "kube-api-access-q6dtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.490391 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.490378    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"
Oct 30 19:15:21.490520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.490403    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc" event={"ID":"b98b2065-c604-4e53-b671-94573cad2963","Type":"ContainerDied","Data":"64a224ccdafb42ef205ee8be191aa5d2e1c9c1f3de9c357179bbbf674a462691"}
Oct 30 19:15:21.492063 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.492046    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5" event={"ID":"842cbd49-4cce-4500-b92b-d9b1446926c2","Type":"ContainerDied","Data":"d31ef08d93e81ecc31134ca622f3c398bae44a5156c265253695095d1e0ad20e"}
Oct 30 19:15:21.492222 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.492094    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"
Oct 30 19:15:21.499547 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.499250    4267 scope.go:122] "RemoveContainer" containerID="a069b89f74e9105999c0baf6180442d0b843d0291290c19892190175aab01ba7"
Oct 30 19:15:21.533536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.532978    4267 scope.go:122] "RemoveContainer" containerID="b8e0fe6308ec7468e7c55aed85eb371ef3e8d509a2dee093fd87e0c37f251545"
Oct 30 19:15:21.587084 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.587053    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6dtb\" (UniqueName: \"kubernetes.io/projected/842cbd49-4cce-4500-b92b-d9b1446926c2-kube-api-access-q6dtb\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.847553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.844543    4267 patch_prober.go:29] interesting pod/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:15:21.847553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.844602    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:15:21.852390 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.851706    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"]
Oct 30 19:15:21.859577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.858581    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4ebeba-29889795-nwdkq"]
Oct 30 19:15:21.986956 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:21.986919    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"]
Oct 30 19:15:22.008580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.008455    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"]
Oct 30 19:15:22.057147 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.054595    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-975cf5-29889795-2gl2z"]
Oct 30 19:15:22.201464 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.201330    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"]
Oct 30 19:15:22.213846 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.213816    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-36d676-29889795-wfbw7"]
Oct 30 19:15:22.268251 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.268175    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"
Oct 30 19:15:22.300444 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.300409    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e1c3881-68a5-4020-adf3-a7399af81b17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1c3881-68a5-4020-adf3-a7399af81b17-maas-api-service-ca\") pod \"0e1c3881-68a5-4020-adf3-a7399af81b17\" (UID: \"0e1c3881-68a5-4020-adf3-a7399af81b17\") "
Oct 30 19:15:22.300718 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.300705    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1c3881-68a5-4020-adf3-a7399af81b17-kube-api-access-vl4tc\" (UniqueName: \"kubernetes.io/projected/0e1c3881-68a5-4020-adf3-a7399af81b17-kube-api-access-vl4tc\") pod \"0e1c3881-68a5-4020-adf3-a7399af81b17\" (UID: \"0e1c3881-68a5-4020-adf3-a7399af81b17\") "
Oct 30 19:15:22.300784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.300753    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1c3881-68a5-4020-adf3-a7399af81b17-maas-api-service-ca" pod "0e1c3881-68a5-4020-adf3-a7399af81b17" (UID: "0e1c3881-68a5-4020-adf3-a7399af81b17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.301117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.301101    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1c3881-68a5-4020-adf3-a7399af81b17-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.302817 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.302798    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1c3881-68a5-4020-adf3-a7399af81b17-kube-api-access-vl4tc" pod "0e1c3881-68a5-4020-adf3-a7399af81b17" (UID: "0e1c3881-68a5-4020-adf3-a7399af81b17"). InnerVolumeSpecName "kube-api-access-vl4tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.402438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.402394    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl4tc\" (UniqueName: \"kubernetes.io/projected/0e1c3881-68a5-4020-adf3-a7399af81b17-kube-api-access-vl4tc\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.403261 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.403227    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"]
Oct 30 19:15:22.431530 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.429117    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-732a3f-29889795-z6nkc"]
Oct 30 19:15:22.504736 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.504706    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"
Oct 30 19:15:22.504736 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.504726    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw" event={"ID":"0e1c3881-68a5-4020-adf3-a7399af81b17","Type":"ContainerDied","Data":"344a49be1c468febd3a79d643b4e14bf4dfeaffe45440a2b38b2103a64c28435"}
Oct 30 19:15:22.505300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.504781    4267 scope.go:122] "RemoveContainer" containerID="7b89d5a9c5f459fae5589d6d57b673e3ccb535b5ac7f83068de284a69af6efdf"
Oct 30 19:15:22.602948 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.602914    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"]
Oct 30 19:15:22.606946 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.606921    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-87ed12-29889795-t6gt5"]
Oct 30 19:15:22.827583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.825588    4267 patch_prober.go:29] interesting pod/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:15:22.827583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.825657    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 19:15:22.893224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.892991    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"]
Oct 30 19:15:22.897420 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:22.897392    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f26b1d-29889795-4rltw"]
Oct 30 19:15:23.297522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:23.296622    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214" path="/var/lib/kubelet/pods/0b63f080-3d3f-46ec-983c-5fe41070c214/volumes"
Oct 30 19:15:23.297522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:23.297260    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17" path="/var/lib/kubelet/pods/0e1c3881-68a5-4020-adf3-a7399af81b17/volumes"
Oct 30 19:15:23.297848 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:23.297832    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" path="/var/lib/kubelet/pods/5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf/volumes"
Oct 30 19:15:23.298835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:23.298817    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2" path="/var/lib/kubelet/pods/842cbd49-4cce-4500-b92b-d9b1446926c2/volumes"
Oct 30 19:15:23.299357 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:23.299343    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98b2065-c604-4e53-b671-94573cad2963" path="/var/lib/kubelet/pods/b98b2065-c604-4e53-b671-94573cad2963/volumes"
Oct 30 19:15:23.299917 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:23.299895    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" path="/var/lib/kubelet/pods/d8c1e641-75e1-4302-82e0-da4f6205f3d8/volumes"
Oct 30 19:15:23.830317 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:23.830280    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:15:23.831680 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:23.831648    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:15:27.359283 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.359214    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" containerID="cri-o://62fb6e2f4e3f5df49fe573bc984b126785eb62f9b395464e61f9085377aad8eb" gracePeriod=45
Oct 30 19:15:27.390989 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.390948    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" containerID="cri-o://a1d20e639c9def53a34dff84c999ee7eec39ff9a16bee5c325241ad36152292e" gracePeriod=45
Oct 30 19:15:27.409699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.409662    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": dial tcp 10.133.0.65:8000: connect: connection refused"
Oct 30 19:15:27.460003 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.459957    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused"
Oct 30 19:15:27.566303 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.566274    4267 generic.go:356] "Generic (PLEG): container finished" podID="7e634b6c-3021-428c-b87d-08912abb84f3" containerID="a1d20e639c9def53a34dff84c999ee7eec39ff9a16bee5c325241ad36152292e" exitCode=0
Oct 30 19:15:27.566596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.566571    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" event={"ID":"7e634b6c-3021-428c-b87d-08912abb84f3","Type":"ContainerDied","Data":"a1d20e639c9def53a34dff84c999ee7eec39ff9a16bee5c325241ad36152292e"}
Oct 30 19:15:27.568406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.568381    4267 generic.go:356] "Generic (PLEG): container finished" podID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerID="62fb6e2f4e3f5df49fe573bc984b126785eb62f9b395464e61f9085377aad8eb" exitCode=0
Oct 30 19:15:27.568529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.568453    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" event={"ID":"c55ad1ee-81e9-444e-a078-7815d3295b4a","Type":"ContainerDied","Data":"62fb6e2f4e3f5df49fe573bc984b126785eb62f9b395464e61f9085377aad8eb"}
Oct 30 19:15:27.695701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.695607    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:15:27.725338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.725312    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:15:27.819963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.819698    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-home\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-home\") pod \"7e634b6c-3021-428c-b87d-08912abb84f3\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") "
Oct 30 19:15:27.819963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.819754    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-home\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-home\") pod \"c55ad1ee-81e9-444e-a078-7815d3295b4a\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") "
Oct 30 19:15:27.819963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.819788    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e634b6c-3021-428c-b87d-08912abb84f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e634b6c-3021-428c-b87d-08912abb84f3-tls-certs\") pod \"7e634b6c-3021-428c-b87d-08912abb84f3\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") "
Oct 30 19:15:27.819963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.819837    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-dshm\") pod \"c55ad1ee-81e9-444e-a078-7815d3295b4a\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") "
Oct 30 19:15:27.820752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.820017    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55ad1ee-81e9-444e-a078-7815d3295b4a-kube-api-access-fcc55\" (UniqueName: \"kubernetes.io/projected/c55ad1ee-81e9-444e-a078-7815d3295b4a-kube-api-access-fcc55\") pod \"c55ad1ee-81e9-444e-a078-7815d3295b4a\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") "
Oct 30 19:15:27.820752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.820090    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-tmp-dir\") pod \"7e634b6c-3021-428c-b87d-08912abb84f3\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") "
Oct 30 19:15:27.820752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.820134    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-model-cache\") pod \"c55ad1ee-81e9-444e-a078-7815d3295b4a\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") "
Oct 30 19:15:27.820752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.820196    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-tmp-dir\") pod \"c55ad1ee-81e9-444e-a078-7815d3295b4a\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") "
Oct 30 19:15:27.820752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.820281    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-dshm\") pod \"7e634b6c-3021-428c-b87d-08912abb84f3\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") "
Oct 30 19:15:27.820752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.820312    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-model-cache\") pod \"7e634b6c-3021-428c-b87d-08912abb84f3\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") "
Oct 30 19:15:27.820752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.820354    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e634b6c-3021-428c-b87d-08912abb84f3-kube-api-access-thlgw\" (UniqueName: \"kubernetes.io/projected/7e634b6c-3021-428c-b87d-08912abb84f3-kube-api-access-thlgw\") pod \"7e634b6c-3021-428c-b87d-08912abb84f3\" (UID: \"7e634b6c-3021-428c-b87d-08912abb84f3\") "
Oct 30 19:15:27.820752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.820427    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55ad1ee-81e9-444e-a078-7815d3295b4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c55ad1ee-81e9-444e-a078-7815d3295b4a-tls-certs\") pod \"c55ad1ee-81e9-444e-a078-7815d3295b4a\" (UID: \"c55ad1ee-81e9-444e-a078-7815d3295b4a\") "
Oct 30 19:15:27.828382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.821263    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-tmp-dir" pod "7e634b6c-3021-428c-b87d-08912abb84f3" (UID: "7e634b6c-3021-428c-b87d-08912abb84f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.828382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.821539    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-home" pod "7e634b6c-3021-428c-b87d-08912abb84f3" (UID: "7e634b6c-3021-428c-b87d-08912abb84f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.828382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.821870    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-model-cache" pod "7e634b6c-3021-428c-b87d-08912abb84f3" (UID: "7e634b6c-3021-428c-b87d-08912abb84f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.828382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.822373    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-home" pod "c55ad1ee-81e9-444e-a078-7815d3295b4a" (UID: "c55ad1ee-81e9-444e-a078-7815d3295b4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.828382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.822389    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-model-cache" pod "c55ad1ee-81e9-444e-a078-7815d3295b4a" (UID: "c55ad1ee-81e9-444e-a078-7815d3295b4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.828382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.822574    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-tmp-dir" pod "c55ad1ee-81e9-444e-a078-7815d3295b4a" (UID: "c55ad1ee-81e9-444e-a078-7815d3295b4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.828382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.826141    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-dshm" pod "7e634b6c-3021-428c-b87d-08912abb84f3" (UID: "7e634b6c-3021-428c-b87d-08912abb84f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.828382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.826180    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55ad1ee-81e9-444e-a078-7815d3295b4a-kube-api-access-fcc55" pod "c55ad1ee-81e9-444e-a078-7815d3295b4a" (UID: "c55ad1ee-81e9-444e-a078-7815d3295b4a"). InnerVolumeSpecName "kube-api-access-fcc55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:27.828382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.826223    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55ad1ee-81e9-444e-a078-7815d3295b4a-tls-certs" pod "c55ad1ee-81e9-444e-a078-7815d3295b4a" (UID: "c55ad1ee-81e9-444e-a078-7815d3295b4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:27.828382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.827240    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e634b6c-3021-428c-b87d-08912abb84f3-tls-certs" pod "7e634b6c-3021-428c-b87d-08912abb84f3" (UID: "7e634b6c-3021-428c-b87d-08912abb84f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:27.829442 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.829414    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e634b6c-3021-428c-b87d-08912abb84f3-kube-api-access-thlgw" pod "7e634b6c-3021-428c-b87d-08912abb84f3" (UID: "7e634b6c-3021-428c-b87d-08912abb84f3"). InnerVolumeSpecName "kube-api-access-thlgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:27.831589 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.831552    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-dshm" pod "c55ad1ee-81e9-444e-a078-7815d3295b4a" (UID: "c55ad1ee-81e9-444e-a078-7815d3295b4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:27.923169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923076    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcc55\" (UniqueName: \"kubernetes.io/projected/c55ad1ee-81e9-444e-a078-7815d3295b4a-kube-api-access-fcc55\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923119    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923153    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923172    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923185    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923197    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923210    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thlgw\" (UniqueName: \"kubernetes.io/projected/7e634b6c-3021-428c-b87d-08912abb84f3-kube-api-access-thlgw\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923241    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55ad1ee-81e9-444e-a078-7815d3295b4a-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923254    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e634b6c-3021-428c-b87d-08912abb84f3-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923265    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923278    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e634b6c-3021-428c-b87d-08912abb84f3-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:27.923536 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:27.923289    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55ad1ee-81e9-444e-a078-7815d3295b4a-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:28.578292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:28.578257    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"
Oct 30 19:15:28.578292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:28.578281    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq" event={"ID":"7e634b6c-3021-428c-b87d-08912abb84f3","Type":"ContainerDied","Data":"8aa1eb13a463d5c128d240081082f48874436d1f5419bf96b9d9f410560835a1"}
Oct 30 19:15:28.578822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:28.578335    4267 scope.go:122] "RemoveContainer" containerID="a1d20e639c9def53a34dff84c999ee7eec39ff9a16bee5c325241ad36152292e"
Oct 30 19:15:28.580304 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:28.580281    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8" event={"ID":"c55ad1ee-81e9-444e-a078-7815d3295b4a","Type":"ContainerDied","Data":"735442d218bf44d2e76dc050c35ddeb09220de651100472cd8cda5d720773c17"}
Oct 30 19:15:28.580447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:28.580315    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"
Oct 30 19:15:28.590583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:28.590560    4267 scope.go:122] "RemoveContainer" containerID="62fb6e2f4e3f5df49fe573bc984b126785eb62f9b395464e61f9085377aad8eb"
Oct 30 19:15:28.745804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:28.745769    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"]
Oct 30 19:15:28.795460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:28.795417    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/simulator-w0-f5763a-kserve-5f774769f-mc7jq"]
Oct 30 19:15:28.859412 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:28.859380    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"]
Oct 30 19:15:28.863656 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:28.863629    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f5763a/distinct-w0-f5763a-kserve-cc589d985-5h8b8"]
Oct 30 19:15:29.223315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:29.223286    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" path="/var/lib/kubelet/pods/7e634b6c-3021-428c-b87d-08912abb84f3/volumes"
Oct 30 19:15:29.223850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:29.223824    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" path="/var/lib/kubelet/pods/c55ad1ee-81e9-444e-a078-7815d3295b4a/volumes"
Oct 30 19:15:31.089343 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:31.089312    4267 scope.go:122] "RemoveContainer" containerID="f7fac6684d17ea0d4912d64c95d0212279f67938c8ab1a99b69235e10f12d0b6"
Oct 30 19:15:31.100361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:31.100339    4267 scope.go:122] "RemoveContainer" containerID="39e73407e39adb2bc3c796f87394bafd9d0c0b32d931ef68ca51f60179a6d40e"
Oct 30 19:15:31.111868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:31.111846    4267 scope.go:122] "RemoveContainer" containerID="26ac5e2a3fb4146ad781e5d254afcd66915a3027c2cce027aab3d122c61e02bf"
Oct 30 19:15:34.250672 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.250635    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"]
Oct 30 19:15:34.251281 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251266    4267 cpu_manager.go:414] "containerMap: removing container" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251282    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251295    4267 cpu_manager.go:414] "containerMap: removing container" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251300    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251307    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b98b2065-c604-4e53-b671-94573cad2963" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251312    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98b2065-c604-4e53-b671-94573cad2963" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251320    4267 cpu_manager.go:414] "containerMap: removing container" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251325    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251330    4267 cpu_manager.go:414] "containerMap: removing container" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251334    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251345    4267 cpu_manager.go:414] "containerMap: removing container" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerName="cleanup"
Oct 30 19:15:34.251350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251349    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251360    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251364    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251372    4267 cpu_manager.go:414] "containerMap: removing container" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251377    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251383    4267 cpu_manager.go:414] "containerMap: removing container" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251388    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251394    4267 cpu_manager.go:414] "containerMap: removing container" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251398    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251410    4267 cpu_manager.go:414] "containerMap: removing container" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251414    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251422    4267 cpu_manager.go:414] "containerMap: removing container" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251426    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251432    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b98b2065-c604-4e53-b671-94573cad2963" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251445    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98b2065-c604-4e53-b671-94573cad2963" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251452    4267 cpu_manager.go:414] "containerMap: removing container" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251456    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251466    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251470    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251478    4267 cpu_manager.go:414] "containerMap: removing container" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251483    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251502    4267 cpu_manager.go:414] "containerMap: removing container" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251506    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251513    4267 cpu_manager.go:414] "containerMap: removing container" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251517    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251523    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251528    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251539    4267 cpu_manager.go:414] "containerMap: removing container" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251545    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251551    4267 cpu_manager.go:414] "containerMap: removing container" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251555    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251561    4267 cpu_manager.go:414] "containerMap: removing container" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251566    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251571    4267 cpu_manager.go:414] "containerMap: removing container" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251575    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251588    4267 cpu_manager.go:414] "containerMap: removing container" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251593    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251598    4267 cpu_manager.go:414] "containerMap: removing container" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251602    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251609    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b98b2065-c604-4e53-b671-94573cad2963" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251613    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98b2065-c604-4e53-b671-94573cad2963" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251720    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerName="cleanup"
Oct 30 19:15:34.251708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251727    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251734    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98b2065-c604-4e53-b671-94573cad2963" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251740    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251745    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251750    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251755    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98b2065-c604-4e53-b671-94573cad2963" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251760    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251765    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251782    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251790    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251796    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251803    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e634b6c-3021-428c-b87d-08912abb84f3" containerName="main"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251809    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251815    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251820    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55ad1ee-81e9-444e-a078-7815d3295b4a" containerName="main"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251825    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251832    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251836    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251842    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251848    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerName="cleanup"
Oct 30 19:15:34.252887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.251852    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca510d4-2aa9-4fb8-b6ff-17ca2f1274bf" containerName="cleanup"
Oct 30 19:15:34.255772 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.255750    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.329646 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.329611    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.329811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.329651    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.329848 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.329813    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.329903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.329844    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.329903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.329878    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.329977 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.329962    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwmz\" (UniqueName: \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-kube-api-access-vzwmz\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.330012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.330003    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.330195 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.330162    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.330275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.330211    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431152    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431205    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431325    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431352    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431380    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431710 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431461    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwmz\" (UniqueName: \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-kube-api-access-vzwmz\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431710 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431527    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431710 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431685    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431852 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431714    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431852 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431724    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.431966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431887    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.432027 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.431985    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.432262 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.432233    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.432606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.432582    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.434301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.434278    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.434886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.434866    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.458754 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.458721    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"]
Oct 30 19:15:34.490163 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.490129    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwmz\" (UniqueName: \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-kube-api-access-vzwmz\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.490324 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.490130    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:34.569162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:34.569074    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:35.036023 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:15:35.035992    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a0bd21_4b46_40dc_99f7_fe2c251fb93d.slice/crio-5a5d387a594d575565f150c4002563f668376dfdd691a0fbe4d5e1f3ddd69f7d WatchSource:0}: Error finding container 5a5d387a594d575565f150c4002563f668376dfdd691a0fbe4d5e1f3ddd69f7d: Status 404 returned error can't find the container with id 5a5d387a594d575565f150c4002563f668376dfdd691a0fbe4d5e1f3ddd69f7d
Oct 30 19:15:35.036284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:35.036259    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"]
Oct 30 19:15:35.038853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:35.038816    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:35.038960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:35.038919    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:35.039026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:35.038966    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:35.651759 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:35.651720    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" event={"ID":"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d","Type":"ContainerStarted","Data":"2cbd31ddda16c23e84ddbf989935cc645f6caf1e6df6e878b0764bff8979398d"}
Oct 30 19:15:35.652335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:35.652302    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" event={"ID":"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d","Type":"ContainerStarted","Data":"5a5d387a594d575565f150c4002563f668376dfdd691a0fbe4d5e1f3ddd69f7d"}
Oct 30 19:15:35.881361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:35.881285    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" podStartSLOduration=1.881263547 podStartE2EDuration="1.881263547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:35.825996693 +0000 UTC m=+2226.713913785" watchObservedRunningTime="2026-10-30 19:15:35.881263547 +0000 UTC m=+2226.769180637"
Oct 30 19:15:36.570038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:36.570001    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:36.571706 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:36.571676    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:15:36.571826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:36.571737    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" podUID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 19:15:37.092129 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.092080    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"]
Oct 30 19:15:37.093014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.092995    4267 cpu_manager.go:414] "containerMap: removing container" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerName="cleanup"
Oct 30 19:15:37.093089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093015    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerName="cleanup"
Oct 30 19:15:37.093089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093054    4267 cpu_manager.go:414] "containerMap: removing container" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerName="cleanup"
Oct 30 19:15:37.093089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093064    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c1e641-75e1-4302-82e0-da4f6205f3d8" containerName="cleanup"
Oct 30 19:15:37.093089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093076    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerName="cleanup"
Oct 30 19:15:37.093089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093084    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerName="cleanup"
Oct 30 19:15:37.093249 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093238    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bdf1c9-20f7-449c-8807-88c5fbe86b35" containerName="cleanup"
Oct 30 19:15:37.093286 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093261    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bb6f2b-8772-4579-88d8-4a347aeee899" containerName="cleanup"
Oct 30 19:15:37.093286 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093274    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1c3881-68a5-4020-adf3-a7399af81b17" containerName="cleanup"
Oct 30 19:15:37.093286 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093282    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="842cbd49-4cce-4500-b92b-d9b1446926c2" containerName="cleanup"
Oct 30 19:15:37.093372 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093295    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98b2065-c604-4e53-b671-94573cad2963" containerName="cleanup"
Oct 30 19:15:37.093372 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093306    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b63f080-3d3f-46ec-983c-5fe41070c214" containerName="cleanup"
Oct 30 19:15:37.093372 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.093316    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaf59c7-05b4-4787-8d74-fb373c68838b" containerName="cleanup"
Oct 30 19:15:37.096031 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.096010    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.146502 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.146457    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"]
Oct 30 19:15:37.150188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.150163    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.153953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.153918    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"]
Oct 30 19:15:37.174392 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.170335    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c9c0534-4hlg5\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.174392 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.170435    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c9c0534-4hlg5\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.174392 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.170468    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgz54\" (UniqueName: \"kubernetes.io/projected/7efeda72-a025-4e51-a54f-15ace119478f-kube-api-access-lgz54\") pod \"maas-api-revoke-sub-3c9c0534-4hlg5\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.191460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.191418    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:37.191999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.191969    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:15:37.264170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.263997    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"]
Oct 30 19:15:37.277207 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.277161    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aef76865-9ghnx\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.277377 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.277355    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aef76865-9ghnx\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.277416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.277406    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c9c0534-4hlg5\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.277488 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.277470    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpnk\" (UniqueName: \"kubernetes.io/projected/94a9ec66-fd5d-411e-a26e-03f17dae3787-kube-api-access-qbpnk\") pod \"maas-api-revoke-sub-aef76865-9ghnx\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.277617 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.277595    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c9c0534-4hlg5\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.277660 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.277639    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgz54\" (UniqueName: \"kubernetes.io/projected/7efeda72-a025-4e51-a54f-15ace119478f-kube-api-access-lgz54\") pod \"maas-api-revoke-sub-3c9c0534-4hlg5\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.279387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.279356    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c9c0534-4hlg5\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.283777 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.283746    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c9c0534-4hlg5\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.339571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.339104    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgz54\" (UniqueName: \"kubernetes.io/projected/7efeda72-a025-4e51-a54f-15ace119478f-kube-api-access-lgz54\") pod \"maas-api-revoke-sub-3c9c0534-4hlg5\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.379101 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.379013    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aef76865-9ghnx\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.379101 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.379086    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpnk\" (UniqueName: \"kubernetes.io/projected/94a9ec66-fd5d-411e-a26e-03f17dae3787-kube-api-access-qbpnk\") pod \"maas-api-revoke-sub-aef76865-9ghnx\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.379339 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.379212    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aef76865-9ghnx\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.379875 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.379850    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aef76865-9ghnx\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.381540 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.381432    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aef76865-9ghnx\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.426517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.425754    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:37.442400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.442366    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpnk\" (UniqueName: \"kubernetes.io/projected/94a9ec66-fd5d-411e-a26e-03f17dae3787-kube-api-access-qbpnk\") pod \"maas-api-revoke-sub-aef76865-9ghnx\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.470243 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.470213    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:37.569829 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.569793    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:15:37.570115 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.570058    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" podUID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 19:15:37.838554 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:15:37.838521    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efeda72_a025_4e51_a54f_15ace119478f.slice/crio-5dcfe3572dc470fc4cdcb9cf047c0d8e85af859c9f34fd2cd1bcf9255ad96824 WatchSource:0}: Error finding container 5dcfe3572dc470fc4cdcb9cf047c0d8e85af859c9f34fd2cd1bcf9255ad96824: Status 404 returned error can't find the container with id 5dcfe3572dc470fc4cdcb9cf047c0d8e85af859c9f34fd2cd1bcf9255ad96824
Oct 30 19:15:37.838925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.838902    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"]
Oct 30 19:15:37.938475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:37.938441    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"]
Oct 30 19:15:37.939951 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:15:37.939924    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a9ec66_fd5d_411e_a26e_03f17dae3787.slice/crio-187abadbd607738ccccbc3e182a06585e16d010a17a8bac72ef48762a4eb0ee9 WatchSource:0}: Error finding container 187abadbd607738ccccbc3e182a06585e16d010a17a8bac72ef48762a4eb0ee9: Status 404 returned error can't find the container with id 187abadbd607738ccccbc3e182a06585e16d010a17a8bac72ef48762a4eb0ee9
Oct 30 19:15:38.570325 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:38.570291    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:15:38.570705 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:38.570339    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" podUID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 19:15:38.679457 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:38.679410    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5" event={"ID":"7efeda72-a025-4e51-a54f-15ace119478f","Type":"ContainerStarted","Data":"2b3748eca58bd46b8a08d99700a7ad1df2db5c7eb1d94703e6c5f0252df41813"}
Oct 30 19:15:38.679647 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:38.679479    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5" event={"ID":"7efeda72-a025-4e51-a54f-15ace119478f","Type":"ContainerStarted","Data":"5dcfe3572dc470fc4cdcb9cf047c0d8e85af859c9f34fd2cd1bcf9255ad96824"}
Oct 30 19:15:38.682030 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:38.682000    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx" event={"ID":"94a9ec66-fd5d-411e-a26e-03f17dae3787","Type":"ContainerStarted","Data":"73a5020ddc79ecd6b4c4b85bf5d794df85ed320c6b40092105226ffdb6992620"}
Oct 30 19:15:38.682146 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:38.682051    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx" event={"ID":"94a9ec66-fd5d-411e-a26e-03f17dae3787","Type":"ContainerStarted","Data":"187abadbd607738ccccbc3e182a06585e16d010a17a8bac72ef48762a4eb0ee9"}
Oct 30 19:15:38.802149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:38.802038    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5" podStartSLOduration=1.8020230160000001 podStartE2EDuration="1.802023016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:38.796035019 +0000 UTC m=+2229.683952108" watchObservedRunningTime="2026-10-30 19:15:38.802023016 +0000 UTC m=+2229.689940112"
Oct 30 19:15:39.570155 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:39.570119    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:15:39.570343 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:39.570172    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" podUID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 19:15:39.690411 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:39.690380    4267 generic.go:356] "Generic (PLEG): container finished" podID="7efeda72-a025-4e51-a54f-15ace119478f" containerID="2b3748eca58bd46b8a08d99700a7ad1df2db5c7eb1d94703e6c5f0252df41813" exitCode=0
Oct 30 19:15:39.690612 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:39.690456    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5" event={"ID":"7efeda72-a025-4e51-a54f-15ace119478f","Type":"ContainerDied","Data":"2b3748eca58bd46b8a08d99700a7ad1df2db5c7eb1d94703e6c5f0252df41813"}
Oct 30 19:15:39.692249 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:39.692226    4267 generic.go:356] "Generic (PLEG): container finished" podID="94a9ec66-fd5d-411e-a26e-03f17dae3787" containerID="73a5020ddc79ecd6b4c4b85bf5d794df85ed320c6b40092105226ffdb6992620" exitCode=0
Oct 30 19:15:39.692381 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:39.692327    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx" event={"ID":"94a9ec66-fd5d-411e-a26e-03f17dae3787","Type":"ContainerDied","Data":"73a5020ddc79ecd6b4c4b85bf5d794df85ed320c6b40092105226ffdb6992620"}
Oct 30 19:15:39.809055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:39.808985    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx" podStartSLOduration=2.808961375 podStartE2EDuration="2.808961375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:38.938830755 +0000 UTC m=+2229.826747846" watchObservedRunningTime="2026-10-30 19:15:39.808961375 +0000 UTC m=+2230.696878464"
Oct 30 19:15:40.570132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:40.570100    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 19:15:40.570299 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:40.570158    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" podUID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 19:15:40.994714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:40.994681    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:41.048704 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.048670    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:41.085419 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.085381    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-openshift-ca\") pod \"94a9ec66-fd5d-411e-a26e-03f17dae3787\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") "
Oct 30 19:15:41.085632 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.085608    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a9ec66-fd5d-411e-a26e-03f17dae3787-kube-api-access-qbpnk\" (UniqueName: \"kubernetes.io/projected/94a9ec66-fd5d-411e-a26e-03f17dae3787-kube-api-access-qbpnk\") pod \"94a9ec66-fd5d-411e-a26e-03f17dae3787\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") "
Oct 30 19:15:41.085719 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.085701    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-xks-ca\") pod \"7efeda72-a025-4e51-a54f-15ace119478f\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") "
Oct 30 19:15:41.085820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.085804    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-xks-ca\") pod \"94a9ec66-fd5d-411e-a26e-03f17dae3787\" (UID: \"94a9ec66-fd5d-411e-a26e-03f17dae3787\") "
Oct 30 19:15:41.085858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.085843    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-openshift-ca" pod "94a9ec66-fd5d-411e-a26e-03f17dae3787" (UID: "94a9ec66-fd5d-411e-a26e-03f17dae3787"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:41.085913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.085881    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7efeda72-a025-4e51-a54f-15ace119478f-kube-api-access-lgz54\" (UniqueName: \"kubernetes.io/projected/7efeda72-a025-4e51-a54f-15ace119478f-kube-api-access-lgz54\") pod \"7efeda72-a025-4e51-a54f-15ace119478f\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") "
Oct 30 19:15:41.085967 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.085920    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-openshift-ca\") pod \"7efeda72-a025-4e51-a54f-15ace119478f\" (UID: \"7efeda72-a025-4e51-a54f-15ace119478f\") "
Oct 30 19:15:41.086365 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.086336    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-openshift-ca" pod "7efeda72-a025-4e51-a54f-15ace119478f" (UID: "7efeda72-a025-4e51-a54f-15ace119478f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:41.086566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.086546    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:41.086694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.086671    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:41.087973 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.087942    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-xks-ca" pod "94a9ec66-fd5d-411e-a26e-03f17dae3787" (UID: "94a9ec66-fd5d-411e-a26e-03f17dae3787"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:41.088413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.088391    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a9ec66-fd5d-411e-a26e-03f17dae3787-kube-api-access-qbpnk" pod "94a9ec66-fd5d-411e-a26e-03f17dae3787" (UID: "94a9ec66-fd5d-411e-a26e-03f17dae3787"). InnerVolumeSpecName "kube-api-access-qbpnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:41.088485 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.088426    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efeda72-a025-4e51-a54f-15ace119478f-kube-api-access-lgz54" pod "7efeda72-a025-4e51-a54f-15ace119478f" (UID: "7efeda72-a025-4e51-a54f-15ace119478f"). InnerVolumeSpecName "kube-api-access-lgz54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:41.089656 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.089635    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-xks-ca" pod "7efeda72-a025-4e51-a54f-15ace119478f" (UID: "7efeda72-a025-4e51-a54f-15ace119478f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:41.188199 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.188167    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efeda72-a025-4e51-a54f-15ace119478f-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:41.188199 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.188198    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a9ec66-fd5d-411e-a26e-03f17dae3787-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:41.188400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.188210    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgz54\" (UniqueName: \"kubernetes.io/projected/7efeda72-a025-4e51-a54f-15ace119478f-kube-api-access-lgz54\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:41.188400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.188220    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbpnk\" (UniqueName: \"kubernetes.io/projected/94a9ec66-fd5d-411e-a26e-03f17dae3787-kube-api-access-qbpnk\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:41.574749 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.574672    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:41.575187 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.575168    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:41.575926 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.575903    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:15:41.708256 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.708226    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"
Oct 30 19:15:41.708452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.708231    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5" event={"ID":"7efeda72-a025-4e51-a54f-15ace119478f","Type":"ContainerDied","Data":"5dcfe3572dc470fc4cdcb9cf047c0d8e85af859c9f34fd2cd1bcf9255ad96824"}
Oct 30 19:15:41.708452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.708358    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dcfe3572dc470fc4cdcb9cf047c0d8e85af859c9f34fd2cd1bcf9255ad96824"
Oct 30 19:15:41.710047 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.710020    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"
Oct 30 19:15:41.710311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.710217    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx" event={"ID":"94a9ec66-fd5d-411e-a26e-03f17dae3787","Type":"ContainerDied","Data":"187abadbd607738ccccbc3e182a06585e16d010a17a8bac72ef48762a4eb0ee9"}
Oct 30 19:15:41.710311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:41.710276    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="187abadbd607738ccccbc3e182a06585e16d010a17a8bac72ef48762a4eb0ee9"
Oct 30 19:15:48.963241 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:48.963200    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"]
Oct 30 19:15:48.963885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:48.963869    4267 cpu_manager.go:414] "containerMap: removing container" podUID="7efeda72-a025-4e51-a54f-15ace119478f" containerName="revoke-keys"
Oct 30 19:15:48.963931 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:48.963886    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7efeda72-a025-4e51-a54f-15ace119478f" containerName="revoke-keys"
Oct 30 19:15:48.963931 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:48.963908    4267 cpu_manager.go:414] "containerMap: removing container" podUID="94a9ec66-fd5d-411e-a26e-03f17dae3787" containerName="revoke-keys"
Oct 30 19:15:48.963931 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:48.963914    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a9ec66-fd5d-411e-a26e-03f17dae3787" containerName="revoke-keys"
Oct 30 19:15:48.964023 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:48.963999    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a9ec66-fd5d-411e-a26e-03f17dae3787" containerName="revoke-keys"
Oct 30 19:15:48.964023 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:48.964017    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="7efeda72-a025-4e51-a54f-15ace119478f" containerName="revoke-keys"
Oct 30 19:15:48.968795 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:48.968776    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.026467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.026431    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"]
Oct 30 19:15:49.070365 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.070315    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:49.070365 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.070350    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:15:49.113074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.113037    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.113264 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.113162    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.113264 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.113233    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb487\" (UniqueName: \"kubernetes.io/projected/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-kube-api-access-mb487\") pod \"maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.215187 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.215077    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.215373 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.215190    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.215373 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.215266    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb487\" (UniqueName: \"kubernetes.io/projected/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-kube-api-access-mb487\") pod \"maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.215972 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.215939    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.230472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.227295    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.275295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.275254    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb487\" (UniqueName: \"kubernetes.io/projected/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-kube-api-access-mb487\") pod \"maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.283226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.283193    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:49.362545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.362097    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"]
Oct 30 19:15:49.370885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.370857    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.461399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.461352    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"]
Oct 30 19:15:49.521745 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.521641    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.522031 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.522012    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.522159 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.522145    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.522377 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.522362    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.522551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.522487    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.522750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.522735    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.522883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.522866    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.522984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.522972    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f3cf07-299e-485e-9bb7-ba2641d40834-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.523189 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.523173    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lht7\" (UniqueName: \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-kube-api-access-8lht7\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.626476    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.626570    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.626675    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.626722    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.626747    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f3cf07-299e-485e-9bb7-ba2641d40834-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.626868    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lht7\" (UniqueName: \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-kube-api-access-8lht7\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.626910    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.626967    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.626992    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.628113    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-data\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.628398    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.629951    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f3cf07-299e-485e-9bb7-ba2641d40834-istiod-ca-cert\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.630204    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-credential-socket\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.630517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.630445    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-certs\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.639600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.639411    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-podinfo\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.690107 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.685195    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-envoy\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.697575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.692137    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-token\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.697575 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.693975    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lht7\" (UniqueName: \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-kube-api-access-8lht7\") pod \"e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") " pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.699888 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.699854    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"]
Oct 30 19:15:49.708306 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:15:49.708279    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd387f2a1_2b01_4a00_9b5b_2fe1d358d58f.slice/crio-67c40cc3752b48e3d7abcf232744302044c5f1c7ed2562d3a3cd7d0801cc8290 WatchSource:0}: Error finding container 67c40cc3752b48e3d7abcf232744302044c5f1c7ed2562d3a3cd7d0801cc8290: Status 404 returned error can't find the container with id 67c40cc3752b48e3d7abcf232744302044c5f1c7ed2562d3a3cd7d0801cc8290
Oct 30 19:15:49.739729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.739697    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:49.839978 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:49.839916    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn" event={"ID":"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f","Type":"ContainerStarted","Data":"67c40cc3752b48e3d7abcf232744302044c5f1c7ed2562d3a3cd7d0801cc8290"}
Oct 30 19:15:50.160646 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:15:50.160599    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:15:50.176669 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:15:50.176628    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f3cf07_299e_485e_9bb7_ba2641d40834.slice/crio-5af6a6f380aec2a0179fcdee6a0b7e6ec5450e3686182ff13dc83671a7dab56b WatchSource:0}: Error finding container 5af6a6f380aec2a0179fcdee6a0b7e6ec5450e3686182ff13dc83671a7dab56b: Status 404 returned error can't find the container with id 5af6a6f380aec2a0179fcdee6a0b7e6ec5450e3686182ff13dc83671a7dab56b
Oct 30 19:15:50.177568 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:50.177540    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"]
Oct 30 19:15:50.179629 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:50.179596    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:50.179690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:50.179664    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:50.179724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:50.179695    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:15:50.854234 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:50.854195    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" event={"ID":"34f3cf07-299e-485e-9bb7-ba2641d40834","Type":"ContainerStarted","Data":"882d8eb23e86bf8429b530cfb89a32bcf8d9f111ca18e3c4d6c96000629fac5c"}
Oct 30 19:15:50.854408 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:50.854267    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" event={"ID":"34f3cf07-299e-485e-9bb7-ba2641d40834","Type":"ContainerStarted","Data":"5af6a6f380aec2a0179fcdee6a0b7e6ec5450e3686182ff13dc83671a7dab56b"}
Oct 30 19:15:50.856175 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:50.856145    4267 generic.go:356] "Generic (PLEG): container finished" podID="d387f2a1-2b01-4a00-9b5b-2fe1d358d58f" containerID="7b822f2742f26cb431563502beff4324c27db883473af00e82f5661756d6bd2f" exitCode=0
Oct 30 19:15:50.856309 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:50.856229    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn" event={"ID":"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f","Type":"ContainerDied","Data":"7b822f2742f26cb431563502beff4324c27db883473af00e82f5661756d6bd2f"}
Oct 30 19:15:51.117587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:51.116553    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" podStartSLOduration=2.116530483 podStartE2EDuration="2.116530483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:51.060044931 +0000 UTC m=+2241.947962030" watchObservedRunningTime="2026-10-30 19:15:51.116530483 +0000 UTC m=+2242.004447572"
Oct 30 19:15:51.744516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:51.742656    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:51.745123 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:51.745089    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 19:15:51.745250 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:51.745155    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 19:15:52.098996 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.098966    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:52.191760 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.191726    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-kube-api-access-mb487\" (UniqueName: \"kubernetes.io/projected/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-kube-api-access-mb487\") pod \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") "
Oct 30 19:15:52.191933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.191887    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-xks-ca\") pod \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") "
Oct 30 19:15:52.192030 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.192001    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-openshift-ca\") pod \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\" (UID: \"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f\") "
Oct 30 19:15:52.192474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.192437    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-openshift-ca" pod "d387f2a1-2b01-4a00-9b5b-2fe1d358d58f" (UID: "d387f2a1-2b01-4a00-9b5b-2fe1d358d58f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:52.192838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.192810    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:52.194035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.194001    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-kube-api-access-mb487" pod "d387f2a1-2b01-4a00-9b5b-2fe1d358d58f" (UID: "d387f2a1-2b01-4a00-9b5b-2fe1d358d58f"). InnerVolumeSpecName "kube-api-access-mb487". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:52.194388 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.194365    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-xks-ca" pod "d387f2a1-2b01-4a00-9b5b-2fe1d358d58f" (UID: "d387f2a1-2b01-4a00-9b5b-2fe1d358d58f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:52.458271 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.442680    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb487\" (UniqueName: \"kubernetes.io/projected/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-kube-api-access-mb487\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:52.458271 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.454377    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:15:52.745573 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.742908    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 19:15:52.745573 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.742977    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 19:15:52.887546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.884568    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"
Oct 30 19:15:52.887546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.886820    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn" event={"ID":"d387f2a1-2b01-4a00-9b5b-2fe1d358d58f","Type":"ContainerDied","Data":"67c40cc3752b48e3d7abcf232744302044c5f1c7ed2562d3a3cd7d0801cc8290"}
Oct 30 19:15:52.887546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:52.886873    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67c40cc3752b48e3d7abcf232744302044c5f1c7ed2562d3a3cd7d0801cc8290"
Oct 30 19:15:53.740742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:53.740700    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 19:15:53.740942 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:53.740782    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 19:15:54.448355 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:54.448317    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"]
Oct 30 19:15:54.496223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:54.496187    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7d059d-4hwr7"]
Oct 30 19:15:54.741070 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:54.740984    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 19:15:54.741224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:54.741065    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 19:15:55.245925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:55.245890    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defecc52-428f-4f15-acef-aabfb38aa43f" path="/var/lib/kubelet/pods/defecc52-428f-4f15-acef-aabfb38aa43f/volumes"
Oct 30 19:15:55.741039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:55.740995    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 19:15:55.741471 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:55.741062    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 19:15:55.854400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:55.854364    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"]
Oct 30 19:15:55.854629 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:55.854574    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy" containerID="cri-o://9491b161f2b0f9d7722191c06ecc70b68a7ef5bf76eab3177fb82971ea58a485" gracePeriod=30
Oct 30 19:15:56.053970 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:56.053887    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"]
Oct 30 19:15:56.054648 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:56.054615    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy" containerID="cri-o://cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1" gracePeriod=30
Oct 30 19:15:56.740958 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:56.740455    4267 patch_prober.go:29] interesting pod/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 19:15:56.740958 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:56.740570    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 19:15:57.647180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:57.647135    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:15:57.647455 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:57.647418    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:15:57.775290 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:57.775251    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:57.775882 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:57.775846    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:15:57.777090 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:15:57.777067    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:16:00.510872 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:00.510831    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"]
Oct 30 19:16:01.026559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.023651    4267 generic.go:356] "Generic (PLEG): container finished" podID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerID="9491b161f2b0f9d7722191c06ecc70b68a7ef5bf76eab3177fb82971ea58a485" exitCode=0
Oct 30 19:16:01.026559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.023815    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" event={"ID":"80d32d01-45a7-4b42-be96-e61fe8ae3545","Type":"ContainerDied","Data":"9491b161f2b0f9d7722191c06ecc70b68a7ef5bf76eab3177fb82971ea58a485"}
Oct 30 19:16:01.293383 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.293349    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.362062    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-kube-api-access-rpkw6\" (UniqueName: \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-kube-api-access-rpkw6\") pod \"80d32d01-45a7-4b42-be96-e61fe8ae3545\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") "
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.362133    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-data\") pod \"80d32d01-45a7-4b42-be96-e61fe8ae3545\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") "
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.362217    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-certs\") pod \"80d32d01-45a7-4b42-be96-e61fe8ae3545\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") "
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.362364    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-envoy\") pod \"80d32d01-45a7-4b42-be96-e61fe8ae3545\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") "
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.362405    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d32d01-45a7-4b42-be96-e61fe8ae3545-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d32d01-45a7-4b42-be96-e61fe8ae3545-istiod-ca-cert\") pod \"80d32d01-45a7-4b42-be96-e61fe8ae3545\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") "
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.362449    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-socket\") pod \"80d32d01-45a7-4b42-be96-e61fe8ae3545\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") "
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.362475    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-credential-socket\") pod \"80d32d01-45a7-4b42-be96-e61fe8ae3545\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") "
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.362543    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-podinfo\") pod \"80d32d01-45a7-4b42-be96-e61fe8ae3545\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") "
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.362582    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-token\" (UniqueName: \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-token\") pod \"80d32d01-45a7-4b42-be96-e61fe8ae3545\" (UID: \"80d32d01-45a7-4b42-be96-e61fe8ae3545\") "
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.363461    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-credential-socket" pod "80d32d01-45a7-4b42-be96-e61fe8ae3545" (UID: "80d32d01-45a7-4b42-be96-e61fe8ae3545"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.364291 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.363832    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d32d01-45a7-4b42-be96-e61fe8ae3545-istiod-ca-cert" pod "80d32d01-45a7-4b42-be96-e61fe8ae3545" (UID: "80d32d01-45a7-4b42-be96-e61fe8ae3545"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:01.367020 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.366634    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-socket" pod "80d32d01-45a7-4b42-be96-e61fe8ae3545" (UID: "80d32d01-45a7-4b42-be96-e61fe8ae3545"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.367020 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.366693    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-certs" pod "80d32d01-45a7-4b42-be96-e61fe8ae3545" (UID: "80d32d01-45a7-4b42-be96-e61fe8ae3545"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.367020 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.366981    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-data" pod "80d32d01-45a7-4b42-be96-e61fe8ae3545" (UID: "80d32d01-45a7-4b42-be96-e61fe8ae3545"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.368183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.368156    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-podinfo" pod "80d32d01-45a7-4b42-be96-e61fe8ae3545" (UID: "80d32d01-45a7-4b42-be96-e61fe8ae3545"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:01.369006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.368982    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-token" pod "80d32d01-45a7-4b42-be96-e61fe8ae3545" (UID: "80d32d01-45a7-4b42-be96-e61fe8ae3545"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:01.369249 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.369130    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-envoy" pod "80d32d01-45a7-4b42-be96-e61fe8ae3545" (UID: "80d32d01-45a7-4b42-be96-e61fe8ae3545"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.369813 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.369786    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-kube-api-access-rpkw6" pod "80d32d01-45a7-4b42-be96-e61fe8ae3545" (UID: "80d32d01-45a7-4b42-be96-e61fe8ae3545"). InnerVolumeSpecName "kube-api-access-rpkw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:01.411665 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.405128    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:16:01.411665 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.405435    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"]
Oct 30 19:16:01.411665 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.405656    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy" containerID="cri-o://09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816" gracePeriod=30
Oct 30 19:16:01.464415 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.464380    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-certs\") pod \"b65fda7a-34ea-4733-ae35-22813f1654b4\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") "
Oct 30 19:16:01.464641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.464479    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-token\" (UniqueName: \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-token\") pod \"b65fda7a-34ea-4733-ae35-22813f1654b4\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") "
Oct 30 19:16:01.464641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.464552    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-credential-socket\") pod \"b65fda7a-34ea-4733-ae35-22813f1654b4\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") "
Oct 30 19:16:01.464870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.464648    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-kube-api-access-4wsr8\" (UniqueName: \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-kube-api-access-4wsr8\") pod \"b65fda7a-34ea-4733-ae35-22813f1654b4\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") "
Oct 30 19:16:01.464870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.464736    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-certs" pod "b65fda7a-34ea-4733-ae35-22813f1654b4" (UID: "b65fda7a-34ea-4733-ae35-22813f1654b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.464870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.464777    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-podinfo\") pod \"b65fda7a-34ea-4733-ae35-22813f1654b4\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") "
Oct 30 19:16:01.464870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.464834    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-data\") pod \"b65fda7a-34ea-4733-ae35-22813f1654b4\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") "
Oct 30 19:16:01.465889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.464875    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-credential-socket" pod "b65fda7a-34ea-4733-ae35-22813f1654b4" (UID: "b65fda7a-34ea-4733-ae35-22813f1654b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.465889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.465194    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65fda7a-34ea-4733-ae35-22813f1654b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65fda7a-34ea-4733-ae35-22813f1654b4-istiod-ca-cert\") pod \"b65fda7a-34ea-4733-ae35-22813f1654b4\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") "
Oct 30 19:16:01.465889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.465272    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-socket\") pod \"b65fda7a-34ea-4733-ae35-22813f1654b4\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") "
Oct 30 19:16:01.465889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.465382    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-envoy\") pod \"b65fda7a-34ea-4733-ae35-22813f1654b4\" (UID: \"b65fda7a-34ea-4733-ae35-22813f1654b4\") "
Oct 30 19:16:01.465889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.465402    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-data" pod "b65fda7a-34ea-4733-ae35-22813f1654b4" (UID: "b65fda7a-34ea-4733-ae35-22813f1654b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.465889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.465553    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-socket" pod "b65fda7a-34ea-4733-ae35-22813f1654b4" (UID: "b65fda7a-34ea-4733-ae35-22813f1654b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.465889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.465657    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65fda7a-34ea-4733-ae35-22813f1654b4-istiod-ca-cert" pod "b65fda7a-34ea-4733-ae35-22813f1654b4" (UID: "b65fda7a-34ea-4733-ae35-22813f1654b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:01.466476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466451    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.466600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466484    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.466600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466518    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.466600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466532    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.466600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466545    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.466600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466557    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.466600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466571    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpkw6\" (UniqueName: \"kubernetes.io/projected/80d32d01-45a7-4b42-be96-e61fe8ae3545-kube-api-access-rpkw6\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.466600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466584    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.466600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466597    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.467039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466609    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.467039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466622    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65fda7a-34ea-4733-ae35-22813f1654b4-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.467039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466634    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.467039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466650    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d32d01-45a7-4b42-be96-e61fe8ae3545-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.467039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.466665    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d32d01-45a7-4b42-be96-e61fe8ae3545-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.467847 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.467822    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-podinfo" pod "b65fda7a-34ea-4733-ae35-22813f1654b4" (UID: "b65fda7a-34ea-4733-ae35-22813f1654b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:01.469625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.469594    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-token" pod "b65fda7a-34ea-4733-ae35-22813f1654b4" (UID: "b65fda7a-34ea-4733-ae35-22813f1654b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:01.469742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.469726    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-kube-api-access-4wsr8" pod "b65fda7a-34ea-4733-ae35-22813f1654b4" (UID: "b65fda7a-34ea-4733-ae35-22813f1654b4"). InnerVolumeSpecName "kube-api-access-4wsr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:01.470827 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.470790    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-envoy" pod "b65fda7a-34ea-4733-ae35-22813f1654b4" (UID: "b65fda7a-34ea-4733-ae35-22813f1654b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:01.568216 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.568176    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.568216 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.568217    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.568797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.568233    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.568797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:01.568249    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wsr8\" (UniqueName: \"kubernetes.io/projected/b65fda7a-34ea-4733-ae35-22813f1654b4-kube-api-access-4wsr8\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:02.124527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.121857    4267 generic.go:356] "Generic (PLEG): container finished" podID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerID="cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1" exitCode=0
Oct 30 19:16:02.124527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.122161    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" event={"ID":"b65fda7a-34ea-4733-ae35-22813f1654b4","Type":"ContainerDied","Data":"cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1"}
Oct 30 19:16:02.124527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.122218    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk" event={"ID":"b65fda7a-34ea-4733-ae35-22813f1654b4","Type":"ContainerDied","Data":"65c55fca4faf02ad2d045f7669a2a37e703d4e05570e9167dda97897645bd681"}
Oct 30 19:16:02.124527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.122248    4267 scope.go:122] "RemoveContainer" containerID="cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1"
Oct 30 19:16:02.124527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.122434    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"
Oct 30 19:16:02.126371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.126344    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"]
Oct 30 19:16:02.135370 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.127389    4267 cpu_manager.go:414] "containerMap: removing container" podUID="d387f2a1-2b01-4a00-9b5b-2fe1d358d58f" containerName="revoke-keys"
Oct 30 19:16:02.135568 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.135546    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d387f2a1-2b01-4a00-9b5b-2fe1d358d58f" containerName="revoke-keys"
Oct 30 19:16:02.135744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.135731    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy"
Oct 30 19:16:02.135826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.135815    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy"
Oct 30 19:16:02.135921 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.135910    4267 cpu_manager.go:414] "containerMap: removing container" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy"
Oct 30 19:16:02.152517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.152470    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"
Oct 30 19:16:02.169053 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.167636    4267 scope.go:122] "RemoveContainer" containerID="cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1"
Oct 30 19:16:02.169053 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:16:02.168662    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1\": container with ID starting with cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1 not found: ID does not exist" containerID="cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1"
Oct 30 19:16:02.169053 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.168702    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1"} err="failed to get container status \"cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1\": rpc error: code = NotFound desc = could not find container \"cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1\": container with ID starting with cd12608614a741583bca5737a04d74599f56231f058d8d776e76b612e158bee1 not found: ID does not exist"
Oct 30 19:16:02.207573 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.207534    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy"
Oct 30 19:16:02.298630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.298598    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="d387f2a1-2b01-4a00-9b5b-2fe1d358d58f" containerName="revoke-keys"
Oct 30 19:16:02.298861 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.298844    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" containerName="istio-proxy"
Oct 30 19:16:02.298962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.298870    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" containerName="istio-proxy"
Oct 30 19:16:02.308867 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.308817    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"]
Oct 30 19:16:02.309068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.308932    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg" event={"ID":"80d32d01-45a7-4b42-be96-e61fe8ae3545","Type":"ContainerDied","Data":"e879715d673b8903cd9f949e1ffaa522e59c62107a24a799d32684e3b46c5fc3"}
Oct 30 19:16:02.309068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.308969    4267 scope.go:122] "RemoveContainer" containerID="9491b161f2b0f9d7722191c06ecc70b68a7ef5bf76eab3177fb82971ea58a485"
Oct 30 19:16:02.309216 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.309159    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:02.378863 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.377946    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:16:02.379152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.379130    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:16:02.399538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.399506    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"]
Oct 30 19:16:02.408690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.408649    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-h9fmg"]
Oct 30 19:16:02.483871 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.483830    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:02.484084 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.484011    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:02.484400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.484374    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7vb\" (UniqueName: \"kubernetes.io/projected/d2b33c8c-899f-4858-b557-c4af9ffb7d23-kube-api-access-cw7vb\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:02.581136 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.581104    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"]
Oct 30 19:16:02.585250 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.585220    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:02.585420 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.585354    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:02.585420 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.585404    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7vb\" (UniqueName: \"kubernetes.io/projected/d2b33c8c-899f-4858-b557-c4af9ffb7d23-kube-api-access-cw7vb\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:02.585959 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.585931    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:02.587792 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.587770    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:02.587964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.587943    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-611053-openshift-default-5cf969d899-6dbrk"]
Oct 30 19:16:02.641946 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.641912    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7vb\" (UniqueName: \"kubernetes.io/projected/d2b33c8c-899f-4858-b557-c4af9ffb7d23-kube-api-access-cw7vb\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:02.719223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:02.719178    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:03.101994 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:16:03.101959    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b33c8c_899f_4858_b557_c4af9ffb7d23.slice/crio-8bc2a04280cda617f4092abc175a61416e3a3f3905137b54f1daf9ce10c3491e WatchSource:0}: Error finding container 8bc2a04280cda617f4092abc175a61416e3a3f3905137b54f1daf9ce10c3491e: Status 404 returned error can't find the container with id 8bc2a04280cda617f4092abc175a61416e3a3f3905137b54f1daf9ce10c3491e
Oct 30 19:16:03.103717 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:03.103689    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"]
Oct 30 19:16:03.166882 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:03.166847    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8" event={"ID":"d2b33c8c-899f-4858-b557-c4af9ffb7d23","Type":"ContainerStarted","Data":"8bc2a04280cda617f4092abc175a61416e3a3f3905137b54f1daf9ce10c3491e"}
Oct 30 19:16:03.223902 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:03.223859    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d32d01-45a7-4b42-be96-e61fe8ae3545" path="/var/lib/kubelet/pods/80d32d01-45a7-4b42-be96-e61fe8ae3545/volumes"
Oct 30 19:16:03.224407 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:03.224393    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65fda7a-34ea-4733-ae35-22813f1654b4" path="/var/lib/kubelet/pods/b65fda7a-34ea-4733-ae35-22813f1654b4/volumes"
Oct 30 19:16:04.179152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:04.179117    4267 generic.go:356] "Generic (PLEG): container finished" podID="d2b33c8c-899f-4858-b557-c4af9ffb7d23" containerID="8cd08142e9fac4b2fcc519e69b15668595259eaa3bba17c0ef19e2347e3704b6" exitCode=0
Oct 30 19:16:04.179683 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:04.179200    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8" event={"ID":"d2b33c8c-899f-4858-b557-c4af9ffb7d23","Type":"ContainerDied","Data":"8cd08142e9fac4b2fcc519e69b15668595259eaa3bba17c0ef19e2347e3704b6"}
Oct 30 19:16:05.343062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.343036    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:05.428517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.428470    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-xks-ca\") pod \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") "
Oct 30 19:16:05.428727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.428559    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b33c8c-899f-4858-b557-c4af9ffb7d23-kube-api-access-cw7vb\" (UniqueName: \"kubernetes.io/projected/d2b33c8c-899f-4858-b557-c4af9ffb7d23-kube-api-access-cw7vb\") pod \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") "
Oct 30 19:16:05.428727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.428604    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-openshift-ca\") pod \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\" (UID: \"d2b33c8c-899f-4858-b557-c4af9ffb7d23\") "
Oct 30 19:16:05.429056 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.429025    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-openshift-ca" pod "d2b33c8c-899f-4858-b557-c4af9ffb7d23" (UID: "d2b33c8c-899f-4858-b557-c4af9ffb7d23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:05.429282 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.429258    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.430815 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.430791    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-xks-ca" pod "d2b33c8c-899f-4858-b557-c4af9ffb7d23" (UID: "d2b33c8c-899f-4858-b557-c4af9ffb7d23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:05.430928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.430879    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b33c8c-899f-4858-b557-c4af9ffb7d23-kube-api-access-cw7vb" pod "d2b33c8c-899f-4858-b557-c4af9ffb7d23" (UID: "d2b33c8c-899f-4858-b557-c4af9ffb7d23"). InnerVolumeSpecName "kube-api-access-cw7vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:05.511546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.511482    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv" podUID="8a7d6069-90d8-484b-b9ad-c98dec33c732" containerName="payload-processing" containerID="cri-o://c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3" gracePeriod=25
Oct 30 19:16:05.530210 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.530181    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b33c8c-899f-4858-b557-c4af9ffb7d23-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.530210 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.530211    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw7vb\" (UniqueName: \"kubernetes.io/projected/d2b33c8c-899f-4858-b557-c4af9ffb7d23-kube-api-access-cw7vb\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.785748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.785721    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:16:05.834953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.834914    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7d6069-90d8-484b-b9ad-c98dec33c732-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7d6069-90d8-484b-b9ad-c98dec33c732-plugins-config-volume\") pod \"8a7d6069-90d8-484b-b9ad-c98dec33c732\" (UID: \"8a7d6069-90d8-484b-b9ad-c98dec33c732\") "
Oct 30 19:16:05.835172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.835139    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7d6069-90d8-484b-b9ad-c98dec33c732-kube-api-access-kb4tr\" (UniqueName: \"kubernetes.io/projected/8a7d6069-90d8-484b-b9ad-c98dec33c732-kube-api-access-kb4tr\") pod \"8a7d6069-90d8-484b-b9ad-c98dec33c732\" (UID: \"8a7d6069-90d8-484b-b9ad-c98dec33c732\") "
Oct 30 19:16:05.835368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.835340    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7d6069-90d8-484b-b9ad-c98dec33c732-plugins-config-volume" pod "8a7d6069-90d8-484b-b9ad-c98dec33c732" (UID: "8a7d6069-90d8-484b-b9ad-c98dec33c732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:05.835748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.835719    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7d6069-90d8-484b-b9ad-c98dec33c732-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.837810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.837782    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7d6069-90d8-484b-b9ad-c98dec33c732-kube-api-access-kb4tr" pod "8a7d6069-90d8-484b-b9ad-c98dec33c732" (UID: "8a7d6069-90d8-484b-b9ad-c98dec33c732"). InnerVolumeSpecName "kube-api-access-kb4tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:05.937297 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:05.937256    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4tr\" (UniqueName: \"kubernetes.io/projected/8a7d6069-90d8-484b-b9ad-c98dec33c732-kube-api-access-kb4tr\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.200541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.198813    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"
Oct 30 19:16:06.200541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.198841    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8" event={"ID":"d2b33c8c-899f-4858-b557-c4af9ffb7d23","Type":"ContainerDied","Data":"8bc2a04280cda617f4092abc175a61416e3a3f3905137b54f1daf9ce10c3491e"}
Oct 30 19:16:06.200541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.198889    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bc2a04280cda617f4092abc175a61416e3a3f3905137b54f1daf9ce10c3491e"
Oct 30 19:16:06.200867 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.200589    4267 generic.go:356] "Generic (PLEG): container finished" podID="8a7d6069-90d8-484b-b9ad-c98dec33c732" containerID="c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3" exitCode=0
Oct 30 19:16:06.200867 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.200632    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"
Oct 30 19:16:06.200867 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.200678    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv" event={"ID":"8a7d6069-90d8-484b-b9ad-c98dec33c732","Type":"ContainerDied","Data":"c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3"}
Oct 30 19:16:06.200867 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.200733    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv" event={"ID":"8a7d6069-90d8-484b-b9ad-c98dec33c732","Type":"ContainerDied","Data":"cf6c14ccf5e446d99edc6f19d71fd779d4722954eafbbfeb041cc8f77e9e7a8a"}
Oct 30 19:16:06.200867 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.200759    4267 scope.go:122] "RemoveContainer" containerID="c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3"
Oct 30 19:16:06.214615 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.214597    4267 scope.go:122] "RemoveContainer" containerID="c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3"
Oct 30 19:16:06.214911 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:16:06.214894    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3\": container with ID starting with c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3 not found: ID does not exist" containerID="c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3"
Oct 30 19:16:06.214984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.214922    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3"} err="failed to get container status \"c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3\": rpc error: code = NotFound desc = could not find container \"c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3\": container with ID starting with c3aac2bffb9448760b2f99aeb11a68437a6779213c96185057c16cc76dbc7df3 not found: ID does not exist"
Oct 30 19:16:06.369517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.368905    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"]
Oct 30 19:16:06.418800 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.418762    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8da6e2-bcd6fcfd5-m52vv"]
Oct 30 19:16:06.730323 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.730294    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:16:06.759864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.759825    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-credential-socket\") pod \"41f3d4ae-3622-413b-9038-debcc28bec24\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") "
Oct 30 19:16:06.760068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.759875    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-data\") pod \"41f3d4ae-3622-413b-9038-debcc28bec24\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") "
Oct 30 19:16:06.760068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.759905    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-socket\") pod \"41f3d4ae-3622-413b-9038-debcc28bec24\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") "
Oct 30 19:16:06.760068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.759969    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-istio-token\" (UniqueName: \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-istio-token\") pod \"41f3d4ae-3622-413b-9038-debcc28bec24\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") "
Oct 30 19:16:06.760068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.760005    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41f3d4ae-3622-413b-9038-debcc28bec24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f3d4ae-3622-413b-9038-debcc28bec24-istio-podinfo\") pod \"41f3d4ae-3622-413b-9038-debcc28bec24\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") "
Oct 30 19:16:06.760300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.760091    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-certs\") pod \"41f3d4ae-3622-413b-9038-debcc28bec24\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") "
Oct 30 19:16:06.760300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.760170    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-credential-socket" pod "41f3d4ae-3622-413b-9038-debcc28bec24" (UID: "41f3d4ae-3622-413b-9038-debcc28bec24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:06.760300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.760196    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-data" pod "41f3d4ae-3622-413b-9038-debcc28bec24" (UID: "41f3d4ae-3622-413b-9038-debcc28bec24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:06.760300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.760205    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-kube-api-access-wdmvz\" (UniqueName: \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-kube-api-access-wdmvz\") pod \"41f3d4ae-3622-413b-9038-debcc28bec24\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") "
Oct 30 19:16:06.760551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.760358    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-envoy\") pod \"41f3d4ae-3622-413b-9038-debcc28bec24\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") "
Oct 30 19:16:06.760551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.760419    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f3d4ae-3622-413b-9038-debcc28bec24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f3d4ae-3622-413b-9038-debcc28bec24-istiod-ca-cert\") pod \"41f3d4ae-3622-413b-9038-debcc28bec24\" (UID: \"41f3d4ae-3622-413b-9038-debcc28bec24\") "
Oct 30 19:16:06.760991 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.760966    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-certs" pod "41f3d4ae-3622-413b-9038-debcc28bec24" (UID: "41f3d4ae-3622-413b-9038-debcc28bec24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:06.760991 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.760982    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-socket" pod "41f3d4ae-3622-413b-9038-debcc28bec24" (UID: "41f3d4ae-3622-413b-9038-debcc28bec24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:06.761322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.761270    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f3d4ae-3622-413b-9038-debcc28bec24-istiod-ca-cert" pod "41f3d4ae-3622-413b-9038-debcc28bec24" (UID: "41f3d4ae-3622-413b-9038-debcc28bec24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:06.761399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.761373    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.761399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.761394    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.761531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.761409    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.761531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.761423    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.762954 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.762909    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-kube-api-access-wdmvz" pod "41f3d4ae-3622-413b-9038-debcc28bec24" (UID: "41f3d4ae-3622-413b-9038-debcc28bec24"). InnerVolumeSpecName "kube-api-access-wdmvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:06.763060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.762986    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41f3d4ae-3622-413b-9038-debcc28bec24-istio-podinfo" pod "41f3d4ae-3622-413b-9038-debcc28bec24" (UID: "41f3d4ae-3622-413b-9038-debcc28bec24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:06.763178 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.763154    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-envoy" pod "41f3d4ae-3622-413b-9038-debcc28bec24" (UID: "41f3d4ae-3622-413b-9038-debcc28bec24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:06.763422 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.763403    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-istio-token" pod "41f3d4ae-3622-413b-9038-debcc28bec24" (UID: "41f3d4ae-3622-413b-9038-debcc28bec24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:06.863422 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.863374    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f3d4ae-3622-413b-9038-debcc28bec24-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.863422 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.863416    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.863422 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.863430    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f3d4ae-3622-413b-9038-debcc28bec24-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.863899 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.863476    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdmvz\" (UniqueName: \"kubernetes.io/projected/41f3d4ae-3622-413b-9038-debcc28bec24-kube-api-access-wdmvz\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.863899 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:06.863522    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f3d4ae-3622-413b-9038-debcc28bec24-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:07.214903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.214859    4267 generic.go:356] "Generic (PLEG): container finished" podID="41f3d4ae-3622-413b-9038-debcc28bec24" containerID="09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816" exitCode=0
Oct 30 19:16:07.215116 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.214960    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"
Oct 30 19:16:07.224942 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.224913    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7d6069-90d8-484b-b9ad-c98dec33c732" path="/var/lib/kubelet/pods/8a7d6069-90d8-484b-b9ad-c98dec33c732/volumes"
Oct 30 19:16:07.225731 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.225705    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" event={"ID":"41f3d4ae-3622-413b-9038-debcc28bec24","Type":"ContainerDied","Data":"09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816"}
Oct 30 19:16:07.225847 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.225739    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4" event={"ID":"41f3d4ae-3622-413b-9038-debcc28bec24","Type":"ContainerDied","Data":"2e16200266888f511c117e336c7bd0294ae8ea9a3698bde3bd647965eacf963b"}
Oct 30 19:16:07.225847 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.225761    4267 scope.go:122] "RemoveContainer" containerID="09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816"
Oct 30 19:16:07.237018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.236995    4267 scope.go:122] "RemoveContainer" containerID="ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44"
Oct 30 19:16:07.247317 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.247293    4267 scope.go:122] "RemoveContainer" containerID="09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816"
Oct 30 19:16:07.247637 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:16:07.247617    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816\": container with ID starting with 09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816 not found: ID does not exist" containerID="09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816"
Oct 30 19:16:07.247701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.247646    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816"} err="failed to get container status \"09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816\": rpc error: code = NotFound desc = could not find container \"09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816\": container with ID starting with 09f87f0c7aab99d985ca43cfcb52dc61a524214e24739a09c42d41bdc8589816 not found: ID does not exist"
Oct 30 19:16:07.247701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.247662    4267 scope.go:122] "RemoveContainer" containerID="ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44"
Oct 30 19:16:07.247939 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:16:07.247917    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44\": container with ID starting with ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44 not found: ID does not exist" containerID="ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44"
Oct 30 19:16:07.247993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.247947    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44"} err="failed to get container status \"ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44\": rpc error: code = NotFound desc = could not find container \"ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44\": container with ID starting with ddaa4e8e274528e3ebc4157fd05a983db1d53af6fc2e2ae8fc03a8f78f8c7c44 not found: ID does not exist"
Oct 30 19:16:07.380730 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.380692    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"]
Oct 30 19:16:07.430371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:07.430343    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7d059d-openshift-default-7ccbcd654c-nmsc4"]
Oct 30 19:16:09.272391 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:09.272359    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" path="/var/lib/kubelet/pods/41f3d4ae-3622-413b-9038-debcc28bec24/volumes"
Oct 30 19:16:11.531276 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:11.531241    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"]
Oct 30 19:16:11.580376 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:11.580344    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ebeba-slxs8"]
Oct 30 19:16:13.224765 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:13.224721    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c79b312b-9ae3-4371-9a77-8cdfc8583c47" path="/var/lib/kubelet/pods/c79b312b-9ae3-4371-9a77-8cdfc8583c47/volumes"
Oct 30 19:16:13.831718 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:13.831681    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"]
Oct 30 19:16:13.938919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:13.938879    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"]
Oct 30 19:16:18.832293 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.832211    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9" podUID="3618fa65-b410-4f76-af60-9e6dd0140f04" containerName="payload-processing" containerID="cri-o://2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925" gracePeriod=25
Oct 30 19:16:18.891368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.891332    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"]
Oct 30 19:16:18.892016 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892001    4267 cpu_manager.go:414] "containerMap: removing container" podUID="8a7d6069-90d8-484b-b9ad-c98dec33c732" containerName="payload-processing"
Oct 30 19:16:18.892079 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892017    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7d6069-90d8-484b-b9ad-c98dec33c732" containerName="payload-processing"
Oct 30 19:16:18.892079 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892052    4267 cpu_manager.go:414] "containerMap: removing container" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy"
Oct 30 19:16:18.892079 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892058    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy"
Oct 30 19:16:18.892079 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892066    4267 cpu_manager.go:414] "containerMap: removing container" podUID="d2b33c8c-899f-4858-b557-c4af9ffb7d23" containerName="revoke-keys"
Oct 30 19:16:18.892079 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892071    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b33c8c-899f-4858-b557-c4af9ffb7d23" containerName="revoke-keys"
Oct 30 19:16:18.892230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892086    4267 cpu_manager.go:414] "containerMap: removing container" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy"
Oct 30 19:16:18.892230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892091    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy"
Oct 30 19:16:18.892230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892181    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7d6069-90d8-484b-b9ad-c98dec33c732" containerName="payload-processing"
Oct 30 19:16:18.892230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892194    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy"
Oct 30 19:16:18.892230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892201    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b33c8c-899f-4858-b557-c4af9ffb7d23" containerName="revoke-keys"
Oct 30 19:16:18.892230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.892209    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f3d4ae-3622-413b-9038-debcc28bec24" containerName="istio-proxy"
Oct 30 19:16:18.895784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.895767    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:18.939620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.939581    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75" podUID="681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef" containerName="payload-pre-processing" containerID="cri-o://7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074" gracePeriod=25
Oct 30 19:16:18.954567 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.954209    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"]
Oct 30 19:16:18.997462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.996740    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:16:18.997462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:18.996902    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:16:19.001897 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.001849    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:19.002057 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.001909    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgzrs\" (UniqueName: \"kubernetes.io/projected/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-kube-api-access-bgzrs\") pod \"maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:19.002210 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.002176    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:19.105546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.104347    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:19.105546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.104545    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:19.105546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.104579    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzrs\" (UniqueName: \"kubernetes.io/projected/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-kube-api-access-bgzrs\") pod \"maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:19.106542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.106124    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:19.112917 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.112872    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:19.150084 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.150053    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:16:19.181540 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.178558    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzrs\" (UniqueName: \"kubernetes.io/projected/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-kube-api-access-bgzrs\") pod \"maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:19.239843 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.239810    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:19.265743 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.265660    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:16:19.307991 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.307719    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3618fa65-b410-4f76-af60-9e6dd0140f04-kube-api-access-94bwb\" (UniqueName: \"kubernetes.io/projected/3618fa65-b410-4f76-af60-9e6dd0140f04-kube-api-access-94bwb\") pod \"3618fa65-b410-4f76-af60-9e6dd0140f04\" (UID: \"3618fa65-b410-4f76-af60-9e6dd0140f04\") "
Oct 30 19:16:19.307991 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.307808    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3618fa65-b410-4f76-af60-9e6dd0140f04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3618fa65-b410-4f76-af60-9e6dd0140f04-plugins-config-volume\") pod \"3618fa65-b410-4f76-af60-9e6dd0140f04\" (UID: \"3618fa65-b410-4f76-af60-9e6dd0140f04\") "
Oct 30 19:16:19.308263 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.308214    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3618fa65-b410-4f76-af60-9e6dd0140f04-plugins-config-volume" pod "3618fa65-b410-4f76-af60-9e6dd0140f04" (UID: "3618fa65-b410-4f76-af60-9e6dd0140f04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:19.309120 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.308709    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3618fa65-b410-4f76-af60-9e6dd0140f04-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:19.310716 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.310060    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3618fa65-b410-4f76-af60-9e6dd0140f04-kube-api-access-94bwb" pod "3618fa65-b410-4f76-af60-9e6dd0140f04" (UID: "3618fa65-b410-4f76-af60-9e6dd0140f04"). InnerVolumeSpecName "kube-api-access-94bwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:19.411922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.409570    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-kube-api-access-5hdt7\" (UniqueName: \"kubernetes.io/projected/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-kube-api-access-5hdt7\") pod \"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef\" (UID: \"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef\") "
Oct 30 19:16:19.411922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.410136    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-plugins-config-volume\") pod \"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef\" (UID: \"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef\") "
Oct 30 19:16:19.411922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.410563    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-plugins-config-volume" pod "681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef" (UID: "681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:19.411922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.411334    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:19.411922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.411358    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94bwb\" (UniqueName: \"kubernetes.io/projected/3618fa65-b410-4f76-af60-9e6dd0140f04-kube-api-access-94bwb\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:19.420184 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.417441    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-kube-api-access-5hdt7" pod "681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef" (UID: "681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef"). InnerVolumeSpecName "kube-api-access-5hdt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:19.483476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.483442    4267 generic.go:356] "Generic (PLEG): container finished" podID="3618fa65-b410-4f76-af60-9e6dd0140f04" containerID="2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925" exitCode=0
Oct 30 19:16:19.483702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.483504    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"
Oct 30 19:16:19.483702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.483528    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9" event={"ID":"3618fa65-b410-4f76-af60-9e6dd0140f04","Type":"ContainerDied","Data":"2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925"}
Oct 30 19:16:19.483702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.483584    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9" event={"ID":"3618fa65-b410-4f76-af60-9e6dd0140f04","Type":"ContainerDied","Data":"e0af0bb05224c1c2eee1da7b9383d33e7231f5f6e05d99cf2c8dae7eb2510952"}
Oct 30 19:16:19.483702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.483612    4267 scope.go:122] "RemoveContainer" containerID="2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925"
Oct 30 19:16:19.485186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.485166    4267 generic.go:356] "Generic (PLEG): container finished" podID="681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef" containerID="7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074" exitCode=0
Oct 30 19:16:19.485321 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.485216    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"
Oct 30 19:16:19.485321 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.485253    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75" event={"ID":"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef","Type":"ContainerDied","Data":"7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074"}
Oct 30 19:16:19.485321 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.485309    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75" event={"ID":"681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef","Type":"ContainerDied","Data":"0c6cf2bab1e848e747d0f3e1b3777d6bf886238e19b001541e48260350ade357"}
Oct 30 19:16:19.496152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.496130    4267 scope.go:122] "RemoveContainer" containerID="2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925"
Oct 30 19:16:19.496649 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:16:19.496477    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925\": container with ID starting with 2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925 not found: ID does not exist" containerID="2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925"
Oct 30 19:16:19.496649 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.496548    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925"} err="failed to get container status \"2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925\": rpc error: code = NotFound desc = could not find container \"2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925\": container with ID starting with 2b5541b151dd2a4305128b3881531b31c7aca7066f8e400b7177d6c77874c925 not found: ID does not exist"
Oct 30 19:16:19.496649 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.496570    4267 scope.go:122] "RemoveContainer" containerID="7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074"
Oct 30 19:16:19.510069 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.510049    4267 scope.go:122] "RemoveContainer" containerID="7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074"
Oct 30 19:16:19.510393 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:16:19.510367    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074\": container with ID starting with 7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074 not found: ID does not exist" containerID="7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074"
Oct 30 19:16:19.510477 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.510404    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074"} err="failed to get container status \"7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074\": rpc error: code = NotFound desc = could not find container \"7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074\": container with ID starting with 7c89df780e468b24ccea33e471a72d38851a0a26664463e65843a4c755b2b074 not found: ID does not exist"
Oct 30 19:16:19.512310 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.512291    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hdt7\" (UniqueName: \"kubernetes.io/projected/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef-kube-api-access-5hdt7\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:19.631534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.631486    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"]
Oct 30 19:16:19.633470 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:16:19.633443    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04bcaa3_94a4_43ed_b307_e7cb51c67b45.slice/crio-19d2254ca9cd46d494b2c0b89056b010851ccf918d9f2a33d7f423d7da398f12 WatchSource:0}: Error finding container 19d2254ca9cd46d494b2c0b89056b010851ccf918d9f2a33d7f423d7da398f12: Status 404 returned error can't find the container with id 19d2254ca9cd46d494b2c0b89056b010851ccf918d9f2a33d7f423d7da398f12
Oct 30 19:16:19.653299 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.653266    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"]
Oct 30 19:16:19.658315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.658283    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-87ed12-7f44f5f455-g2v75"]
Oct 30 19:16:19.778614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.778577    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"]
Oct 30 19:16:19.782701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:19.782673    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-87ed12-7c6d4bc7f-8n8t9"]
Oct 30 19:16:20.196229 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:16:20.196192    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:16:20.497241 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.497199    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh" event={"ID":"f04bcaa3-94a4-43ed-b307-e7cb51c67b45","Type":"ContainerStarted","Data":"dc5deb38742c79d334c74216fa0c7e167069a279fa3ccb1893d75cd7a2cbf927"}
Oct 30 19:16:20.497442 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.497271    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh" event={"ID":"f04bcaa3-94a4-43ed-b307-e7cb51c67b45","Type":"ContainerStarted","Data":"19d2254ca9cd46d494b2c0b89056b010851ccf918d9f2a33d7f423d7da398f12"}
Oct 30 19:16:20.612000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.611935    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh" podStartSLOduration=2.611919958 podStartE2EDuration="2.611919958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:20.611434576 +0000 UTC m=+2271.499351657" watchObservedRunningTime="2026-10-30 19:16:20.611919958 +0000 UTC m=+2271.499837046"
Oct 30 19:16:20.975431 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.975391    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"]
Oct 30 19:16:20.976852 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.976810    4267 cpu_manager.go:414] "containerMap: removing container" podUID="3618fa65-b410-4f76-af60-9e6dd0140f04" containerName="payload-processing"
Oct 30 19:16:20.976852 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.976849    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3618fa65-b410-4f76-af60-9e6dd0140f04" containerName="payload-processing"
Oct 30 19:16:20.977050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.976882    4267 cpu_manager.go:414] "containerMap: removing container" podUID="681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef" containerName="payload-pre-processing"
Oct 30 19:16:20.977050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.976892    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef" containerName="payload-pre-processing"
Oct 30 19:16:20.977136 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.977120    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="3618fa65-b410-4f76-af60-9e6dd0140f04" containerName="payload-processing"
Oct 30 19:16:20.977191 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.977141    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef" containerName="payload-pre-processing"
Oct 30 19:16:20.981810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:20.981432    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.026258 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.026224    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"]
Oct 30 19:16:21.146105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.146071    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.146279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.146205    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.146279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.146259    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6dxq\" (UniqueName: \"kubernetes.io/projected/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-kube-api-access-n6dxq\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.221744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.221713    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3618fa65-b410-4f76-af60-9e6dd0140f04" path="/var/lib/kubelet/pods/3618fa65-b410-4f76-af60-9e6dd0140f04/volumes"
Oct 30 19:16:21.222185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.222171    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef" path="/var/lib/kubelet/pods/681a58de-ae69-4ea5-a6c0-4ee14aa3a3ef/volumes"
Oct 30 19:16:21.247443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.247364    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.247604 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.247460    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.247604 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.247503    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6dxq\" (UniqueName: \"kubernetes.io/projected/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-kube-api-access-n6dxq\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.248121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.248091    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.249695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.249674    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.304943 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.304915    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6dxq\" (UniqueName: \"kubernetes.io/projected/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-kube-api-access-n6dxq\") pod \"maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.507739 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.507655    4267 generic.go:356] "Generic (PLEG): container finished" podID="f04bcaa3-94a4-43ed-b307-e7cb51c67b45" containerID="dc5deb38742c79d334c74216fa0c7e167069a279fa3ccb1893d75cd7a2cbf927" exitCode=0
Oct 30 19:16:21.507899 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.507734    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh" event={"ID":"f04bcaa3-94a4-43ed-b307-e7cb51c67b45","Type":"ContainerDied","Data":"dc5deb38742c79d334c74216fa0c7e167069a279fa3ccb1893d75cd7a2cbf927"}
Oct 30 19:16:21.599380 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.599348    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:21.981233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:21.981206    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"]
Oct 30 19:16:21.981813 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:16:21.981782    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd345e1_5dd4_414b_8f48_2b0ca0f8df50.slice/crio-deac1a8a6fa4788001ebe6f323270f7c0289e028b948b113cbdb028cb3db7281 WatchSource:0}: Error finding container deac1a8a6fa4788001ebe6f323270f7c0289e028b948b113cbdb028cb3db7281: Status 404 returned error can't find the container with id deac1a8a6fa4788001ebe6f323270f7c0289e028b948b113cbdb028cb3db7281
Oct 30 19:16:22.518350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.518232    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4" event={"ID":"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50","Type":"ContainerStarted","Data":"039eb835bea300c76fdb4d9ccd9b04ba7177578c1179f4b50b87b2550d5aab75"}
Oct 30 19:16:22.518350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.518314    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4" event={"ID":"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50","Type":"ContainerStarted","Data":"deac1a8a6fa4788001ebe6f323270f7c0289e028b948b113cbdb028cb3db7281"}
Oct 30 19:16:22.632645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.632580    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4" podStartSLOduration=2.632558719 podStartE2EDuration="2.632558719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:22.631244181 +0000 UTC m=+2273.519161268" watchObservedRunningTime="2026-10-30 19:16:22.632558719 +0000 UTC m=+2273.520475811"
Oct 30 19:16:22.681233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.681205    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:22.773689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.773592    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-xks-ca\") pod \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") "
Oct 30 19:16:22.773689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.773674    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-openshift-ca\") pod \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") "
Oct 30 19:16:22.773964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.773942    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-kube-api-access-bgzrs\" (UniqueName: \"kubernetes.io/projected/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-kube-api-access-bgzrs\") pod \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\" (UID: \"f04bcaa3-94a4-43ed-b307-e7cb51c67b45\") "
Oct 30 19:16:22.774050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.774028    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-openshift-ca" pod "f04bcaa3-94a4-43ed-b307-e7cb51c67b45" (UID: "f04bcaa3-94a4-43ed-b307-e7cb51c67b45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:22.774446 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.774426    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:22.775836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.775803    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-xks-ca" pod "f04bcaa3-94a4-43ed-b307-e7cb51c67b45" (UID: "f04bcaa3-94a4-43ed-b307-e7cb51c67b45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:22.776058 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.776035    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-kube-api-access-bgzrs" pod "f04bcaa3-94a4-43ed-b307-e7cb51c67b45" (UID: "f04bcaa3-94a4-43ed-b307-e7cb51c67b45"). InnerVolumeSpecName "kube-api-access-bgzrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:22.875387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.875353    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgzrs\" (UniqueName: \"kubernetes.io/projected/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-kube-api-access-bgzrs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:22.875387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:22.875384    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04bcaa3-94a4-43ed-b307-e7cb51c67b45-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.527238 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:23.527143    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"
Oct 30 19:16:23.527238 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:23.527147    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh" event={"ID":"f04bcaa3-94a4-43ed-b307-e7cb51c67b45","Type":"ContainerDied","Data":"19d2254ca9cd46d494b2c0b89056b010851ccf918d9f2a33d7f423d7da398f12"}
Oct 30 19:16:23.527826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:23.527290    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19d2254ca9cd46d494b2c0b89056b010851ccf918d9f2a33d7f423d7da398f12"
Oct 30 19:16:23.529364 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:23.529337    4267 generic.go:356] "Generic (PLEG): container finished" podID="bdd345e1-5dd4-414b-8f48-2b0ca0f8df50" containerID="039eb835bea300c76fdb4d9ccd9b04ba7177578c1179f4b50b87b2550d5aab75" exitCode=0
Oct 30 19:16:23.529542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:23.529430    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4" event={"ID":"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50","Type":"ContainerDied","Data":"039eb835bea300c76fdb4d9ccd9b04ba7177578c1179f4b50b87b2550d5aab75"}
Oct 30 19:16:24.713312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:24.713286    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:24.801785 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:24.801694    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-kube-api-access-n6dxq\" (UniqueName: \"kubernetes.io/projected/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-kube-api-access-n6dxq\") pod \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") "
Oct 30 19:16:24.801785 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:24.801764    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-openshift-ca\") pod \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") "
Oct 30 19:16:24.801989 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:24.801853    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-xks-ca\") pod \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\" (UID: \"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50\") "
Oct 30 19:16:24.802237 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:24.802205    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-openshift-ca" pod "bdd345e1-5dd4-414b-8f48-2b0ca0f8df50" (UID: "bdd345e1-5dd4-414b-8f48-2b0ca0f8df50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:24.802555 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:24.802536    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:24.804044 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:24.804016    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-xks-ca" pod "bdd345e1-5dd4-414b-8f48-2b0ca0f8df50" (UID: "bdd345e1-5dd4-414b-8f48-2b0ca0f8df50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:24.804155 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:24.804133    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-kube-api-access-n6dxq" pod "bdd345e1-5dd4-414b-8f48-2b0ca0f8df50" (UID: "bdd345e1-5dd4-414b-8f48-2b0ca0f8df50"). InnerVolumeSpecName "kube-api-access-n6dxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:24.903904 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:24.903846    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6dxq\" (UniqueName: \"kubernetes.io/projected/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-kube-api-access-n6dxq\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:24.903904 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:24.903891    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:25.545778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:25.545679    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"
Oct 30 19:16:25.545778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:25.545711    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4" event={"ID":"bdd345e1-5dd4-414b-8f48-2b0ca0f8df50","Type":"ContainerDied","Data":"deac1a8a6fa4788001ebe6f323270f7c0289e028b948b113cbdb028cb3db7281"}
Oct 30 19:16:25.545778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:25.545761    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deac1a8a6fa4788001ebe6f323270f7c0289e028b948b113cbdb028cb3db7281"
Oct 30 19:16:30.353521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:30.352994    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"]
Oct 30 19:16:30.437511 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:30.437433    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"]
Oct 30 19:16:35.354179 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.354135    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7" podUID="d62500e8-c362-4892-a8b8-003da67a4112" containerName="payload-processing" containerID="cri-o://f27ffe7b1b52db165fcf3964600fe55499470366cad8dc39fae0ac3609632f86" gracePeriod=25
Oct 30 19:16:35.438012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.437963    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5" podUID="3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe" containerName="payload-pre-processing" containerID="cri-o://6cc7dfb7315181b229ae984ec8edf23eeedc5934bbedf2e55a11fcec9cd1a782" gracePeriod=25
Oct 30 19:16:35.680637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.680603    4267 generic.go:356] "Generic (PLEG): container finished" podID="d62500e8-c362-4892-a8b8-003da67a4112" containerID="f27ffe7b1b52db165fcf3964600fe55499470366cad8dc39fae0ac3609632f86" exitCode=0
Oct 30 19:16:35.680831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.680659    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7" event={"ID":"d62500e8-c362-4892-a8b8-003da67a4112","Type":"ContainerDied","Data":"f27ffe7b1b52db165fcf3964600fe55499470366cad8dc39fae0ac3609632f86"}
Oct 30 19:16:35.683007 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.682970    4267 generic.go:356] "Generic (PLEG): container finished" podID="3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe" containerID="6cc7dfb7315181b229ae984ec8edf23eeedc5934bbedf2e55a11fcec9cd1a782" exitCode=0
Oct 30 19:16:35.683130 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.683012    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5" event={"ID":"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe","Type":"ContainerDied","Data":"6cc7dfb7315181b229ae984ec8edf23eeedc5934bbedf2e55a11fcec9cd1a782"}
Oct 30 19:16:35.737741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.737706    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:16:35.770183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.770155    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:16:35.859545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.859468    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62500e8-c362-4892-a8b8-003da67a4112-kube-api-access-w7zsh\" (UniqueName: \"kubernetes.io/projected/d62500e8-c362-4892-a8b8-003da67a4112-kube-api-access-w7zsh\") pod \"d62500e8-c362-4892-a8b8-003da67a4112\" (UID: \"d62500e8-c362-4892-a8b8-003da67a4112\") "
Oct 30 19:16:35.859697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.859556    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62500e8-c362-4892-a8b8-003da67a4112-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62500e8-c362-4892-a8b8-003da67a4112-plugins-config-volume\") pod \"d62500e8-c362-4892-a8b8-003da67a4112\" (UID: \"d62500e8-c362-4892-a8b8-003da67a4112\") "
Oct 30 19:16:35.859738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.859714    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-plugins-config-volume\") pod \"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe\" (UID: \"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe\") "
Oct 30 19:16:35.859870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.859848    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-kube-api-access-5cshd\" (UniqueName: \"kubernetes.io/projected/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-kube-api-access-5cshd\") pod \"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe\" (UID: \"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe\") "
Oct 30 19:16:35.860102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.860074    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62500e8-c362-4892-a8b8-003da67a4112-plugins-config-volume" pod "d62500e8-c362-4892-a8b8-003da67a4112" (UID: "d62500e8-c362-4892-a8b8-003da67a4112"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:35.860226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.860104    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-plugins-config-volume" pod "3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe" (UID: "3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:35.860479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.860459    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62500e8-c362-4892-a8b8-003da67a4112-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:35.860479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.860478    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:35.861783 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.861756    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62500e8-c362-4892-a8b8-003da67a4112-kube-api-access-w7zsh" pod "d62500e8-c362-4892-a8b8-003da67a4112" (UID: "d62500e8-c362-4892-a8b8-003da67a4112"). InnerVolumeSpecName "kube-api-access-w7zsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:35.862091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.862069    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-kube-api-access-5cshd" pod "3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe" (UID: "3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe"). InnerVolumeSpecName "kube-api-access-5cshd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:35.962378 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.962331    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cshd\" (UniqueName: \"kubernetes.io/projected/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe-kube-api-access-5cshd\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:35.962378 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:35.962369    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7zsh\" (UniqueName: \"kubernetes.io/projected/d62500e8-c362-4892-a8b8-003da67a4112-kube-api-access-w7zsh\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:16:36.691949 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:36.691919    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"
Oct 30 19:16:36.692473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:36.691946    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7" event={"ID":"d62500e8-c362-4892-a8b8-003da67a4112","Type":"ContainerDied","Data":"0a5861110ec3849722d5158077533384ff81431c68b6ce9da94b3472ac5799c3"}
Oct 30 19:16:36.692473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:36.692007    4267 scope.go:122] "RemoveContainer" containerID="f27ffe7b1b52db165fcf3964600fe55499470366cad8dc39fae0ac3609632f86"
Oct 30 19:16:36.693977 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:36.693950    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5" event={"ID":"3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe","Type":"ContainerDied","Data":"086dbdb6dcb3b69881c7c0d9af67d61e9224a5e413dc60d903bcbb71c6afcbc8"}
Oct 30 19:16:36.693977 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:36.693970    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"
Oct 30 19:16:36.704483 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:36.704432    4267 scope.go:122] "RemoveContainer" containerID="6cc7dfb7315181b229ae984ec8edf23eeedc5934bbedf2e55a11fcec9cd1a782"
Oct 30 19:16:36.914524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:36.913553    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"]
Oct 30 19:16:36.928112 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:36.924156    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24e88a-597b894b45-mflg7"]
Oct 30 19:16:37.044571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:37.038555    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"]
Oct 30 19:16:37.073663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:37.073618    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24e88a-77db8ccb67-vrvt5"]
Oct 30 19:16:37.359564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:37.358546    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe" path="/var/lib/kubelet/pods/3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe/volumes"
Oct 30 19:16:37.359564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:37.359451    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62500e8-c362-4892-a8b8-003da67a4112" path="/var/lib/kubelet/pods/d62500e8-c362-4892-a8b8-003da67a4112/volumes"
Oct 30 19:16:49.049993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:49.049902    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"]
Oct 30 19:16:49.096858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:49.096822    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebe4e56-jf4gh"]
Oct 30 19:16:49.224018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:16:49.223958    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636edd24-8b5b-4e31-b8b1-fce5b6fd03c0" path="/var/lib/kubelet/pods/636edd24-8b5b-4e31-b8b1-fce5b6fd03c0/volumes"
Oct 30 19:16:50.159943 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:16:50.159901    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:17:03.603682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:03.603645    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"]
Oct 30 19:17:03.652836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:03.652799    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8da6e2-c95wn"]
Oct 30 19:17:05.222269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:05.222235    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d387f2a1-2b01-4a00-9b5b-2fe1d358d58f" path="/var/lib/kubelet/pods/d387f2a1-2b01-4a00-9b5b-2fe1d358d58f/volumes"
Oct 30 19:17:09.039485 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:09.039444    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"]
Oct 30 19:17:09.040135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:09.039710    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy" containerID="cri-o://9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77" gracePeriod=30
Oct 30 19:17:09.053702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:09.053663    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"]
Oct 30 19:17:09.053954 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:09.053923    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy" containerID="cri-o://943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2" gracePeriod=30
Oct 30 19:17:09.684054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:09.684015    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"]
Oct 30 19:17:09.688149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:09.688117    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f26b1d-n7zt9"]
Oct 30 19:17:10.791742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:10.791703    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"]
Oct 30 19:17:10.792269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:10.791929    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy" containerID="cri-o://877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777" gracePeriod=30
Oct 30 19:17:11.224211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:11.224172    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c7a105-02f2-496e-ac2d-73f64d42e7bd" path="/var/lib/kubelet/pods/06c7a105-02f2-496e-ac2d-73f64d42e7bd/volumes"
Oct 30 19:17:12.889263 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.889226    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"]
Oct 30 19:17:12.889835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.889819    4267 cpu_manager.go:414] "containerMap: removing container" podUID="bdd345e1-5dd4-414b-8f48-2b0ca0f8df50" containerName="revoke-keys"
Oct 30 19:17:12.889895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.889836    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd345e1-5dd4-414b-8f48-2b0ca0f8df50" containerName="revoke-keys"
Oct 30 19:17:12.889895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.889856    4267 cpu_manager.go:414] "containerMap: removing container" podUID="d62500e8-c362-4892-a8b8-003da67a4112" containerName="payload-processing"
Oct 30 19:17:12.889895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.889861    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62500e8-c362-4892-a8b8-003da67a4112" containerName="payload-processing"
Oct 30 19:17:12.889895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.889882    4267 cpu_manager.go:414] "containerMap: removing container" podUID="f04bcaa3-94a4-43ed-b307-e7cb51c67b45" containerName="revoke-keys"
Oct 30 19:17:12.889895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.889890    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04bcaa3-94a4-43ed-b307-e7cb51c67b45" containerName="revoke-keys"
Oct 30 19:17:12.890068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.889923    4267 cpu_manager.go:414] "containerMap: removing container" podUID="3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe" containerName="payload-pre-processing"
Oct 30 19:17:12.890068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.889928    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe" containerName="payload-pre-processing"
Oct 30 19:17:12.890068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.890029    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf014bc-a4db-4af3-bd2f-14f9ad23f6fe" containerName="payload-pre-processing"
Oct 30 19:17:12.890068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.890040    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd345e1-5dd4-414b-8f48-2b0ca0f8df50" containerName="revoke-keys"
Oct 30 19:17:12.890068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.890048    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62500e8-c362-4892-a8b8-003da67a4112" containerName="payload-processing"
Oct 30 19:17:12.890068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.890055    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04bcaa3-94a4-43ed-b307-e7cb51c67b45" containerName="revoke-keys"
Oct 30 19:17:12.894749 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.894732    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:12.951884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.951850    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"]
Oct 30 19:17:12.996329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.996293    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:12.996329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:12.996305    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:17:13.021304 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.021272    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:13.021532 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.021342    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:13.021608 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.021528    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62rk\" (UniqueName: \"kubernetes.io/projected/06c66545-4766-4956-ae0f-b416735b36ab-kube-api-access-k62rk\") pod \"maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:13.122969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.122931    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62rk\" (UniqueName: \"kubernetes.io/projected/06c66545-4766-4956-ae0f-b416735b36ab-kube-api-access-k62rk\") pod \"maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:13.123198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.123070    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:13.123198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.123144    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:13.124014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.123987    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:13.125654 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.125626    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:13.180288 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.180208    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62rk\" (UniqueName: \"kubernetes.io/projected/06c66545-4766-4956-ae0f-b416735b36ab-kube-api-access-k62rk\") pod \"maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:13.208560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.208527    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:13.599817 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:17:13.599782    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c66545_4766_4956_ae0f_b416735b36ab.slice/crio-f5afe4a4ec5975a11d344d16f873ddab32eb52872f373cddd1017e84329fe9ff WatchSource:0}: Error finding container f5afe4a4ec5975a11d344d16f873ddab32eb52872f373cddd1017e84329fe9ff: Status 404 returned error can't find the container with id f5afe4a4ec5975a11d344d16f873ddab32eb52872f373cddd1017e84329fe9ff
Oct 30 19:17:13.600749 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:13.600721    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"]
Oct 30 19:17:14.114395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.114359    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl" event={"ID":"06c66545-4766-4956-ae0f-b416735b36ab","Type":"ContainerStarted","Data":"81dbc80e7b649abdef77f81fcb8c8bdf321a7e4eca4762d983950787b5bf9205"}
Oct 30 19:17:14.114817 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.114439    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl" event={"ID":"06c66545-4766-4956-ae0f-b416735b36ab","Type":"ContainerStarted","Data":"f5afe4a4ec5975a11d344d16f873ddab32eb52872f373cddd1017e84329fe9ff"}
Oct 30 19:17:14.344820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.344782    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:17:14.427978 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.427911    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:17:14.444063 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444030    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-socket\") pod \"fc708cda-e0e7-4c57-99e9-81c0d158077a\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") "
Oct 30 19:17:14.444268 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444092    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-credential-socket\") pod \"fc708cda-e0e7-4c57-99e9-81c0d158077a\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") "
Oct 30 19:17:14.444268 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444170    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-token\" (UniqueName: \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-token\") pod \"fc708cda-e0e7-4c57-99e9-81c0d158077a\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") "
Oct 30 19:17:14.444268 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444237    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc708cda-e0e7-4c57-99e9-81c0d158077a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc708cda-e0e7-4c57-99e9-81c0d158077a-istiod-ca-cert\") pod \"fc708cda-e0e7-4c57-99e9-81c0d158077a\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") "
Oct 30 19:17:14.444443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444318    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-podinfo\") pod \"fc708cda-e0e7-4c57-99e9-81c0d158077a\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") "
Oct 30 19:17:14.444443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444320    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-socket" pod "fc708cda-e0e7-4c57-99e9-81c0d158077a" (UID: "fc708cda-e0e7-4c57-99e9-81c0d158077a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.444443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444364    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-data\") pod \"fc708cda-e0e7-4c57-99e9-81c0d158077a\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") "
Oct 30 19:17:14.444618 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444448    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-credential-socket" pod "fc708cda-e0e7-4c57-99e9-81c0d158077a" (UID: "fc708cda-e0e7-4c57-99e9-81c0d158077a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.444618 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444476    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-certs\") pod \"fc708cda-e0e7-4c57-99e9-81c0d158077a\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") "
Oct 30 19:17:14.444618 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444549    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-kube-api-access-qcrwz\" (UniqueName: \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-kube-api-access-qcrwz\") pod \"fc708cda-e0e7-4c57-99e9-81c0d158077a\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") "
Oct 30 19:17:14.444767 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444622    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-envoy\") pod \"fc708cda-e0e7-4c57-99e9-81c0d158077a\" (UID: \"fc708cda-e0e7-4c57-99e9-81c0d158077a\") "
Oct 30 19:17:14.444767 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444702    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-data" pod "fc708cda-e0e7-4c57-99e9-81c0d158077a" (UID: "fc708cda-e0e7-4c57-99e9-81c0d158077a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.444926 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444901    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-certs" pod "fc708cda-e0e7-4c57-99e9-81c0d158077a" (UID: "fc708cda-e0e7-4c57-99e9-81c0d158077a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.445008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.444990    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc708cda-e0e7-4c57-99e9-81c0d158077a-istiod-ca-cert" pod "fc708cda-e0e7-4c57-99e9-81c0d158077a" (UID: "fc708cda-e0e7-4c57-99e9-81c0d158077a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:14.445235 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.445213    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc708cda-e0e7-4c57-99e9-81c0d158077a-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.445346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.445238    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.445346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.445253    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.445346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.445269    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.445346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.445286    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.446934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.446913    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-podinfo" pod "fc708cda-e0e7-4c57-99e9-81c0d158077a" (UID: "fc708cda-e0e7-4c57-99e9-81c0d158077a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:14.447200 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.447174    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-envoy" pod "fc708cda-e0e7-4c57-99e9-81c0d158077a" (UID: "fc708cda-e0e7-4c57-99e9-81c0d158077a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.447394 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.447377    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-token" pod "fc708cda-e0e7-4c57-99e9-81c0d158077a" (UID: "fc708cda-e0e7-4c57-99e9-81c0d158077a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.447488 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.447406    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-kube-api-access-qcrwz" pod "fc708cda-e0e7-4c57-99e9-81c0d158077a" (UID: "fc708cda-e0e7-4c57-99e9-81c0d158077a"). InnerVolumeSpecName "kube-api-access-qcrwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.466656 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.466597    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl" podStartSLOduration=2.466577556 podStartE2EDuration="2.466577556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:14.234528726 +0000 UTC m=+2325.122445816" watchObservedRunningTime="2026-10-30 19:17:14.466577556 +0000 UTC m=+2325.354494646"
Oct 30 19:17:14.545981 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.545940    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-credential-socket\") pod \"df5c2d3f-4778-470b-8cc3-3f96f7529302\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") "
Oct 30 19:17:14.546188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.546056    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-kube-api-access-czskk\" (UniqueName: \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-kube-api-access-czskk\") pod \"df5c2d3f-4778-470b-8cc3-3f96f7529302\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") "
Oct 30 19:17:14.546188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.546160    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-token\" (UniqueName: \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-token\") pod \"df5c2d3f-4778-470b-8cc3-3f96f7529302\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") "
Oct 30 19:17:14.546306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.546200    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-podinfo\") pod \"df5c2d3f-4778-470b-8cc3-3f96f7529302\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") "
Oct 30 19:17:14.546306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.546282    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-data\") pod \"df5c2d3f-4778-470b-8cc3-3f96f7529302\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") "
Oct 30 19:17:14.546415 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.546354    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-socket\") pod \"df5c2d3f-4778-470b-8cc3-3f96f7529302\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") "
Oct 30 19:17:14.546415 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.546393    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-certs\") pod \"df5c2d3f-4778-470b-8cc3-3f96f7529302\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") "
Oct 30 19:17:14.546533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.546423    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-envoy\") pod \"df5c2d3f-4778-470b-8cc3-3f96f7529302\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") "
Oct 30 19:17:14.546533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.546462    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5c2d3f-4778-470b-8cc3-3f96f7529302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5c2d3f-4778-470b-8cc3-3f96f7529302-istiod-ca-cert\") pod \"df5c2d3f-4778-470b-8cc3-3f96f7529302\" (UID: \"df5c2d3f-4778-470b-8cc3-3f96f7529302\") "
Oct 30 19:17:14.546927 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.546901    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-credential-socket" pod "df5c2d3f-4778-470b-8cc3-3f96f7529302" (UID: "df5c2d3f-4778-470b-8cc3-3f96f7529302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.547207 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.547188    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcrwz\" (UniqueName: \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-kube-api-access-qcrwz\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.547354 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.547339    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.547452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.547431    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.547586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.547572    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.547682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.547669    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc708cda-e0e7-4c57-99e9-81c0d158077a-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.547763 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.547479    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-data" pod "df5c2d3f-4778-470b-8cc3-3f96f7529302" (UID: "df5c2d3f-4778-470b-8cc3-3f96f7529302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.547842 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.547735    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-socket" pod "df5c2d3f-4778-470b-8cc3-3f96f7529302" (UID: "df5c2d3f-4778-470b-8cc3-3f96f7529302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.547955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.547920    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5c2d3f-4778-470b-8cc3-3f96f7529302-istiod-ca-cert" pod "df5c2d3f-4778-470b-8cc3-3f96f7529302" (UID: "df5c2d3f-4778-470b-8cc3-3f96f7529302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:14.548208 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.548181    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-certs" pod "df5c2d3f-4778-470b-8cc3-3f96f7529302" (UID: "df5c2d3f-4778-470b-8cc3-3f96f7529302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.550776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.550741    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-token" pod "df5c2d3f-4778-470b-8cc3-3f96f7529302" (UID: "df5c2d3f-4778-470b-8cc3-3f96f7529302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.551160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.551136    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-kube-api-access-czskk" pod "df5c2d3f-4778-470b-8cc3-3f96f7529302" (UID: "df5c2d3f-4778-470b-8cc3-3f96f7529302"). InnerVolumeSpecName "kube-api-access-czskk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.552845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.552819    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-envoy" pod "df5c2d3f-4778-470b-8cc3-3f96f7529302" (UID: "df5c2d3f-4778-470b-8cc3-3f96f7529302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.554394 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.554369    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-podinfo" pod "df5c2d3f-4778-470b-8cc3-3f96f7529302" (UID: "df5c2d3f-4778-470b-8cc3-3f96f7529302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:14.649334 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.649299    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.649334 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.649331    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.649334 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.649341    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.649689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.649349    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5c2d3f-4778-470b-8cc3-3f96f7529302-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.649689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.649358    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czskk\" (UniqueName: \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-kube-api-access-czskk\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.649689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.649367    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.649689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.649377    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.649689 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:14.649385    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5c2d3f-4778-470b-8cc3-3f96f7529302-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:15.125144 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.125102    4267 generic.go:356] "Generic (PLEG): container finished" podID="06c66545-4766-4956-ae0f-b416735b36ab" containerID="81dbc80e7b649abdef77f81fcb8c8bdf321a7e4eca4762d983950787b5bf9205" exitCode=0
Oct 30 19:17:15.125687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.125180    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl" event={"ID":"06c66545-4766-4956-ae0f-b416735b36ab","Type":"ContainerDied","Data":"81dbc80e7b649abdef77f81fcb8c8bdf321a7e4eca4762d983950787b5bf9205"}
Oct 30 19:17:15.126885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.126859    4267 generic.go:356] "Generic (PLEG): container finished" podID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerID="943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2" exitCode=0
Oct 30 19:17:15.127039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.126915    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"
Oct 30 19:17:15.127039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.126933    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" event={"ID":"df5c2d3f-4778-470b-8cc3-3f96f7529302","Type":"ContainerDied","Data":"943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2"}
Oct 30 19:17:15.127039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.126979    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6" event={"ID":"df5c2d3f-4778-470b-8cc3-3f96f7529302","Type":"ContainerDied","Data":"8edeb36182dd59e2a482b121f86f79dad5cbff8dfc7993343d829a5283f1ad7d"}
Oct 30 19:17:15.127039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.127010    4267 scope.go:122] "RemoveContainer" containerID="943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2"
Oct 30 19:17:15.128825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.128798    4267 generic.go:356] "Generic (PLEG): container finished" podID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerID="9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77" exitCode=0
Oct 30 19:17:15.128958 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.128844    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"
Oct 30 19:17:15.128958 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.128896    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" event={"ID":"fc708cda-e0e7-4c57-99e9-81c0d158077a","Type":"ContainerDied","Data":"9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77"}
Oct 30 19:17:15.129146 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.128937    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7" event={"ID":"fc708cda-e0e7-4c57-99e9-81c0d158077a","Type":"ContainerDied","Data":"b95cac5aee91c47138555182e3850e6f14780f1283dfa2b5c2ba11e86028fd57"}
Oct 30 19:17:15.144798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.144769    4267 scope.go:122] "RemoveContainer" containerID="943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2"
Oct 30 19:17:15.145906 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:15.145879    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2\": container with ID starting with 943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2 not found: ID does not exist" containerID="943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2"
Oct 30 19:17:15.146006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.145917    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2"} err="failed to get container status \"943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2\": rpc error: code = NotFound desc = could not find container \"943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2\": container with ID starting with 943ba555986ba1cc43a66577968a3a22a953c385363fd7b3023278d279ba3eb2 not found: ID does not exist"
Oct 30 19:17:15.146006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.145936    4267 scope.go:122] "RemoveContainer" containerID="9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77"
Oct 30 19:17:15.156596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.156476    4267 scope.go:122] "RemoveContainer" containerID="9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77"
Oct 30 19:17:15.156852 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:15.156828    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77\": container with ID starting with 9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77 not found: ID does not exist" containerID="9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77"
Oct 30 19:17:15.156955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.156858    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77"} err="failed to get container status \"9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77\": rpc error: code = NotFound desc = could not find container \"9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77\": container with ID starting with 9c803a0d2f2c0250bd0e69f54040a0dcb2fdc6ee953005b983603b66c424aa77 not found: ID does not exist"
Oct 30 19:17:15.358529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.358459    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"]
Oct 30 19:17:15.362426 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.362397    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-tvft6"]
Oct 30 19:17:15.479141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.479084    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"]
Oct 30 19:17:15.484249 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:15.484211    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5763a-openshift-default-545765b89-8wbp7"]
Oct 30 19:17:16.084302 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.084270    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:17:16.143335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.143297    4267 generic.go:356] "Generic (PLEG): container finished" podID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerID="877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777" exitCode=0
Oct 30 19:17:16.143824 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.143371    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"
Oct 30 19:17:16.143824 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.143380    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" event={"ID":"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5","Type":"ContainerDied","Data":"877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777"}
Oct 30 19:17:16.143824 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.143469    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw" event={"ID":"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5","Type":"ContainerDied","Data":"5c4a0d8f8bf6699f53e71b3752653c93abb132895c00a2fd42bb8955cc2fe9ad"}
Oct 30 19:17:16.143824 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.143510    4267 scope.go:122] "RemoveContainer" containerID="877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777"
Oct 30 19:17:16.155211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.155184    4267 scope.go:122] "RemoveContainer" containerID="5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d"
Oct 30 19:17:16.166912 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.166885    4267 scope.go:122] "RemoveContainer" containerID="877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777"
Oct 30 19:17:16.167273 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:16.167240    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777\": container with ID starting with 877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777 not found: ID does not exist" containerID="877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777"
Oct 30 19:17:16.167346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.167293    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777"} err="failed to get container status \"877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777\": rpc error: code = NotFound desc = could not find container \"877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777\": container with ID starting with 877208631fbb0fb660e4b2d9a57ab86a3a5130651b3fc4659c2fcbf0c5a74777 not found: ID does not exist"
Oct 30 19:17:16.167346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.167324    4267 scope.go:122] "RemoveContainer" containerID="5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d"
Oct 30 19:17:16.167688 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:16.167667    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d\": container with ID starting with 5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d not found: ID does not exist" containerID="5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d"
Oct 30 19:17:16.167773 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.167696    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d"} err="failed to get container status \"5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d\": rpc error: code = NotFound desc = could not find container \"5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d\": container with ID starting with 5194c54e89daf4082f5d64a6143c68bd3d68f90d4f1295b38fadd259534a174d not found: ID does not exist"
Oct 30 19:17:16.172049 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172025    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-data\") pod \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") "
Oct 30 19:17:16.172162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172098    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-token\" (UniqueName: \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-token\") pod \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") "
Oct 30 19:17:16.172212 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172168    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-socket\") pod \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") "
Oct 30 19:17:16.172212 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172202    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-envoy\") pod \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") "
Oct 30 19:17:16.172318 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172222    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-kube-api-access-6tbds\" (UniqueName: \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-kube-api-access-6tbds\") pod \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") "
Oct 30 19:17:16.172318 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172292    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-data" pod "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" (UID: "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:16.172318 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172310    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-podinfo\") pod \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") "
Oct 30 19:17:16.172473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172338    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-certs\") pod \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") "
Oct 30 19:17:16.172473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172395    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istiod-ca-cert\") pod \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") "
Oct 30 19:17:16.172473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172462    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-socket" pod "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" (UID: "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:16.172637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172477    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-credential-socket\") pod \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\" (UID: \"a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5\") "
Oct 30 19:17:16.172696 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172681    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-credential-socket" pod "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" (UID: "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:16.172950 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.172918    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-certs" pod "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" (UID: "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:16.173211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.173187    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.173276 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.173218    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.173276 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.173233    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.173276 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.173246    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.173465 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.173430    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istiod-ca-cert" pod "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" (UID: "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:16.174611 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.174586    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-kube-api-access-6tbds" pod "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" (UID: "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5"). InnerVolumeSpecName "kube-api-access-6tbds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:16.174912 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.174887    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-envoy" pod "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" (UID: "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:16.175009 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.174984    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-podinfo" pod "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" (UID: "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:16.175364 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.175343    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-token" pod "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" (UID: "a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:16.275537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.275368    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.275537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.275408    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.275537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.275424    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tbds\" (UniqueName: \"kubernetes.io/projected/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-kube-api-access-6tbds\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.275537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.275448    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.275537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.275465    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.317329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.317259    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:16.376865 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.376818    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-xks-ca\") pod \"06c66545-4766-4956-ae0f-b416735b36ab\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") "
Oct 30 19:17:16.377048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.376909    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c66545-4766-4956-ae0f-b416735b36ab-kube-api-access-k62rk\" (UniqueName: \"kubernetes.io/projected/06c66545-4766-4956-ae0f-b416735b36ab-kube-api-access-k62rk\") pod \"06c66545-4766-4956-ae0f-b416735b36ab\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") "
Oct 30 19:17:16.377048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.376952    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-openshift-ca\") pod \"06c66545-4766-4956-ae0f-b416735b36ab\" (UID: \"06c66545-4766-4956-ae0f-b416735b36ab\") "
Oct 30 19:17:16.377348 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.377319    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-openshift-ca" pod "06c66545-4766-4956-ae0f-b416735b36ab" (UID: "06c66545-4766-4956-ae0f-b416735b36ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:16.377577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.377555    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.379146 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.379124    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-xks-ca" pod "06c66545-4766-4956-ae0f-b416735b36ab" (UID: "06c66545-4766-4956-ae0f-b416735b36ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:16.379244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.379224    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c66545-4766-4956-ae0f-b416735b36ab-kube-api-access-k62rk" pod "06c66545-4766-4956-ae0f-b416735b36ab" (UID: "06c66545-4766-4956-ae0f-b416735b36ab"). InnerVolumeSpecName "kube-api-access-k62rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:16.479445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.479405    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c66545-4766-4956-ae0f-b416735b36ab-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.479445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.479448    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62rk\" (UniqueName: \"kubernetes.io/projected/06c66545-4766-4956-ae0f-b416735b36ab-kube-api-access-k62rk\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:16.569822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.569731    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"]
Oct 30 19:17:16.573503 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:16.573467    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8da6e2-openshift-default-586f58bd67-lthzw"]
Oct 30 19:17:17.154068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.154031    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"
Oct 30 19:17:17.154068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.154042    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl" event={"ID":"06c66545-4766-4956-ae0f-b416735b36ab","Type":"ContainerDied","Data":"f5afe4a4ec5975a11d344d16f873ddab32eb52872f373cddd1017e84329fe9ff"}
Oct 30 19:17:17.154615 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.154099    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5afe4a4ec5975a11d344d16f873ddab32eb52872f373cddd1017e84329fe9ff"
Oct 30 19:17:17.224062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.224023    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" path="/var/lib/kubelet/pods/a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5/volumes"
Oct 30 19:17:17.224809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.224790    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" path="/var/lib/kubelet/pods/df5c2d3f-4778-470b-8cc3-3f96f7529302/volumes"
Oct 30 19:17:17.225451 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.225433    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" path="/var/lib/kubelet/pods/fc708cda-e0e7-4c57-99e9-81c0d158077a/volumes"
Oct 30 19:17:17.555009 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.554966    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"]
Oct 30 19:17:17.555854 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.555829    4267 cpu_manager.go:414] "containerMap: removing container" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy"
Oct 30 19:17:17.555854 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.555853    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy"
Oct 30 19:17:17.556026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.555875    4267 cpu_manager.go:414] "containerMap: removing container" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy"
Oct 30 19:17:17.556026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.555883    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy"
Oct 30 19:17:17.556026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.555895    4267 cpu_manager.go:414] "containerMap: removing container" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy"
Oct 30 19:17:17.556026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.555903    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy"
Oct 30 19:17:17.556026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.555928    4267 cpu_manager.go:414] "containerMap: removing container" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy"
Oct 30 19:17:17.556026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.555936    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy"
Oct 30 19:17:17.556026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.555959    4267 cpu_manager.go:414] "containerMap: removing container" podUID="06c66545-4766-4956-ae0f-b416735b36ab" containerName="revoke-keys"
Oct 30 19:17:17.556026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.555966    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c66545-4766-4956-ae0f-b416735b36ab" containerName="revoke-keys"
Oct 30 19:17:17.556425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.556117    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc708cda-e0e7-4c57-99e9-81c0d158077a" containerName="istio-proxy"
Oct 30 19:17:17.556425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.556130    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5c2d3f-4778-470b-8cc3-3f96f7529302" containerName="istio-proxy"
Oct 30 19:17:17.556425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.556146    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy"
Oct 30 19:17:17.556425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.556158    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c66545-4766-4956-ae0f-b416735b36ab" containerName="revoke-keys"
Oct 30 19:17:17.561571 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.561547    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:17.614671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.614634    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"]
Oct 30 19:17:17.624703 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.624670    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:17.659254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.659221    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:17:17.663756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.663727    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"]
Oct 30 19:17:17.678383 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.678352    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"]
Oct 30 19:17:17.704903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.704870    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-903a82-gmscw\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:17.705108 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.705004    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-903a82-gmscw\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:17.705108 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.705063    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75s9q\" (UniqueName: \"kubernetes.io/projected/92408e1f-b0f2-4d10-9899-71656d977ed6-kube-api-access-75s9q\") pod \"maas-api-revoke-keys-e2e-api-b-903a82-gmscw\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:17.806621 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.806431    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75s9q\" (UniqueName: \"kubernetes.io/projected/92408e1f-b0f2-4d10-9899-71656d977ed6-kube-api-access-75s9q\") pod \"maas-api-revoke-keys-e2e-api-b-903a82-gmscw\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:17.806835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.806811    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-903a82-gmscw\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:17.806944 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.806921    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-903a82-gmscw\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:17.807682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.807658    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-903a82-gmscw\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:17.810055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.810028    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-903a82-gmscw\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:17.863136 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.863105    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75s9q\" (UniqueName: \"kubernetes.io/projected/92408e1f-b0f2-4d10-9899-71656d977ed6-kube-api-access-75s9q\") pod \"maas-api-revoke-keys-e2e-api-b-903a82-gmscw\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:17.876196 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:17.876168    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:18.262579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:18.262551    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"]
Oct 30 19:17:18.265905 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:17:18.265875    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92408e1f_b0f2_4d10_9899_71656d977ed6.slice/crio-14fbf9cfe1909a73c7e81c84583dbf10ddba754a4566baded5e41ea7a24bb980 WatchSource:0}: Error finding container 14fbf9cfe1909a73c7e81c84583dbf10ddba754a4566baded5e41ea7a24bb980: Status 404 returned error can't find the container with id 14fbf9cfe1909a73c7e81c84583dbf10ddba754a4566baded5e41ea7a24bb980
Oct 30 19:17:19.174968 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:19.174919    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw" event={"ID":"92408e1f-b0f2-4d10-9899-71656d977ed6","Type":"ContainerStarted","Data":"4a2b9468728d03a900920cf1b7c0d20e6d3324f1679d33b9839ca3d8653a5ff6"}
Oct 30 19:17:19.175184 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:19.174988    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw" event={"ID":"92408e1f-b0f2-4d10-9899-71656d977ed6","Type":"ContainerStarted","Data":"14fbf9cfe1909a73c7e81c84583dbf10ddba754a4566baded5e41ea7a24bb980"}
Oct 30 19:17:20.164842 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:20.164805    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:17:20.183640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:20.183598    4267 generic.go:356] "Generic (PLEG): container finished" podID="92408e1f-b0f2-4d10-9899-71656d977ed6" containerID="4a2b9468728d03a900920cf1b7c0d20e6d3324f1679d33b9839ca3d8653a5ff6" exitCode=0
Oct 30 19:17:20.183836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:20.183691    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw" event={"ID":"92408e1f-b0f2-4d10-9899-71656d977ed6","Type":"ContainerDied","Data":"4a2b9468728d03a900920cf1b7c0d20e6d3324f1679d33b9839ca3d8653a5ff6"}
Oct 30 19:17:21.178706 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.178675    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"]
Oct 30 19:17:21.182455 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.182428    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae6a138d-hb9d4"]
Oct 30 19:17:21.230334 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.230285    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd345e1-5dd4-414b-8f48-2b0ca0f8df50" path="/var/lib/kubelet/pods/bdd345e1-5dd4-414b-8f48-2b0ca0f8df50/volumes"
Oct 30 19:17:21.487795 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.487389    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:21.678502 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.678457    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-xks-ca\") pod \"92408e1f-b0f2-4d10-9899-71656d977ed6\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") "
Oct 30 19:17:21.678690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.678575    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92408e1f-b0f2-4d10-9899-71656d977ed6-kube-api-access-75s9q\" (UniqueName: \"kubernetes.io/projected/92408e1f-b0f2-4d10-9899-71656d977ed6-kube-api-access-75s9q\") pod \"92408e1f-b0f2-4d10-9899-71656d977ed6\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") "
Oct 30 19:17:21.678690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.678680    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-openshift-ca\") pod \"92408e1f-b0f2-4d10-9899-71656d977ed6\" (UID: \"92408e1f-b0f2-4d10-9899-71656d977ed6\") "
Oct 30 19:17:21.679022 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.678998    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-openshift-ca" pod "92408e1f-b0f2-4d10-9899-71656d977ed6" (UID: "92408e1f-b0f2-4d10-9899-71656d977ed6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:21.679275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.679258    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:21.680682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.680650    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-xks-ca" pod "92408e1f-b0f2-4d10-9899-71656d977ed6" (UID: "92408e1f-b0f2-4d10-9899-71656d977ed6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:21.680760 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.680710    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92408e1f-b0f2-4d10-9899-71656d977ed6-kube-api-access-75s9q" pod "92408e1f-b0f2-4d10-9899-71656d977ed6" (UID: "92408e1f-b0f2-4d10-9899-71656d977ed6"). InnerVolumeSpecName "kube-api-access-75s9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:21.780718 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.780645    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92408e1f-b0f2-4d10-9899-71656d977ed6-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:21.780718 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:21.780677    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75s9q\" (UniqueName: \"kubernetes.io/projected/92408e1f-b0f2-4d10-9899-71656d977ed6-kube-api-access-75s9q\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:22.200504 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:22.200461    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"
Oct 30 19:17:22.200903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:22.200517    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw" event={"ID":"92408e1f-b0f2-4d10-9899-71656d977ed6","Type":"ContainerDied","Data":"14fbf9cfe1909a73c7e81c84583dbf10ddba754a4566baded5e41ea7a24bb980"}
Oct 30 19:17:22.200903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:22.200563    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14fbf9cfe1909a73c7e81c84583dbf10ddba754a4566baded5e41ea7a24bb980"
Oct 30 19:17:22.623067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:22.622984    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"]
Oct 30 19:17:22.628756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:22.628732    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-87ed12-h7jk8"]
Oct 30 19:17:23.220878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:23.220846    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b33c8c-899f-4858-b557-c4af9ffb7d23" path="/var/lib/kubelet/pods/d2b33c8c-899f-4858-b557-c4af9ffb7d23/volumes"
Oct 30 19:17:24.344012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:24.343977    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"]
Oct 30 19:17:29.003868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:29.003824    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"]
Oct 30 19:17:29.111283 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:29.111245    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"]
Oct 30 19:17:29.344988 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:29.344902    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj" podUID="81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa" containerName="payload-processing" containerID="cri-o://b06dee0f36c748139250a94141aa7a998af464415d329827dfa0b8fc2b6d87ee" gracePeriod=25
Oct 30 19:17:29.411326 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:29.411289    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"]
Oct 30 19:17:29.411968 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:29.411583    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerName="istio-proxy" containerID="cri-o://92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606" gracePeriod=30
Oct 30 19:17:31.281885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:31.281851    4267 scope.go:122] "RemoveContainer" containerID="201165b380506633435774868608a5029be8081206fcf3e9614f5150e0d64053"
Oct 30 19:17:31.292217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:31.292196    4267 scope.go:122] "RemoveContainer" containerID="aef33eab280b5904ed2ed25246d5513fa7bbf4af9dc4eb99d6d7750f359fc4d8"
Oct 30 19:17:31.891874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:31.891839    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"]
Oct 30 19:17:31.892086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:31.892062    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q" podUID="078a0cc5-96c9-49b4-bd07-91cc514a556e" containerName="maas-api" containerID="cri-o://e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c" gracePeriod=30
Oct 30 19:17:32.250480 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.250448    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:17:32.316322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.316278    4267 generic.go:356] "Generic (PLEG): container finished" podID="078a0cc5-96c9-49b4-bd07-91cc514a556e" containerID="e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c" exitCode=0
Oct 30 19:17:32.316845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.316334    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"
Oct 30 19:17:32.316845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.316332    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q" event={"ID":"078a0cc5-96c9-49b4-bd07-91cc514a556e","Type":"ContainerDied","Data":"e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c"}
Oct 30 19:17:32.316845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.316390    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q" event={"ID":"078a0cc5-96c9-49b4-bd07-91cc514a556e","Type":"ContainerDied","Data":"31d958dfcb2b10e8801b2d49321644c847845e307d8ee916158638de761d4e56"}
Oct 30 19:17:32.316845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.316426    4267 scope.go:122] "RemoveContainer" containerID="e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c"
Oct 30 19:17:32.329866 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.329835    4267 scope.go:122] "RemoveContainer" containerID="e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c"
Oct 30 19:17:32.330196 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:32.330177    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c\": container with ID starting with e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c not found: ID does not exist" containerID="e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c"
Oct 30 19:17:32.330296 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.330207    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c"} err="failed to get container status \"e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c\": rpc error: code = NotFound desc = could not find container \"e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c\": container with ID starting with e26d54e6ce993fae013a2b4f210a947bf71b6eb6e7e34f60847207469e38e38c not found: ID does not exist"
Oct 30 19:17:32.382606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.382572    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078a0cc5-96c9-49b4-bd07-91cc514a556e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078a0cc5-96c9-49b4-bd07-91cc514a556e-openshift-service-ca\") pod \"078a0cc5-96c9-49b4-bd07-91cc514a556e\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") "
Oct 30 19:17:32.382798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.382703    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078a0cc5-96c9-49b4-bd07-91cc514a556e-kube-api-access-vbb4x\" (UniqueName: \"kubernetes.io/projected/078a0cc5-96c9-49b4-bd07-91cc514a556e-kube-api-access-vbb4x\") pod \"078a0cc5-96c9-49b4-bd07-91cc514a556e\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") "
Oct 30 19:17:32.382879 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.382866    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-maas-api-tls\") pod \"078a0cc5-96c9-49b4-bd07-91cc514a556e\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") "
Oct 30 19:17:32.382975 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.382917    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-metrics-certs\") pod \"078a0cc5-96c9-49b4-bd07-91cc514a556e\" (UID: \"078a0cc5-96c9-49b4-bd07-91cc514a556e\") "
Oct 30 19:17:32.405089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.403121    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-metrics-certs" pod "078a0cc5-96c9-49b4-bd07-91cc514a556e" (UID: "078a0cc5-96c9-49b4-bd07-91cc514a556e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:32.423515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.422111    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078a0cc5-96c9-49b4-bd07-91cc514a556e-openshift-service-ca" pod "078a0cc5-96c9-49b4-bd07-91cc514a556e" (UID: "078a0cc5-96c9-49b4-bd07-91cc514a556e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:32.484541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.484481    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-maas-api-tls" pod "078a0cc5-96c9-49b4-bd07-91cc514a556e" (UID: "078a0cc5-96c9-49b4-bd07-91cc514a556e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:32.488572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.485868    4267 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078a0cc5-96c9-49b4-bd07-91cc514a556e-openshift-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:32.488572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.486474    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-maas-api-tls\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:32.488572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.486659    4267 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078a0cc5-96c9-49b4-bd07-91cc514a556e-metrics-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:32.489108 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.489065    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078a0cc5-96c9-49b4-bd07-91cc514a556e-kube-api-access-vbb4x" pod "078a0cc5-96c9-49b4-bd07-91cc514a556e" (UID: "078a0cc5-96c9-49b4-bd07-91cc514a556e"). InnerVolumeSpecName "kube-api-access-vbb4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:32.591140 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.591093    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbb4x\" (UniqueName: \"kubernetes.io/projected/078a0cc5-96c9-49b4-bd07-91cc514a556e-kube-api-access-vbb4x\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:32.685674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.685629    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" containerID="cri-o://8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d" gracePeriod=45
Oct 30 19:17:32.698794 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.698759    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main" containerID="cri-o://c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9" gracePeriod=45
Oct 30 19:17:32.729632 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.729541    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Oct 30 19:17:32.739078 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.739045    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"]
Oct 30 19:17:32.788411 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:32.788370    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-24e88a-5b859db68c-ltz2q"]
Oct 30 19:17:33.088645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.087456    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:17:33.120580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.114704    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:17:33.168538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168484    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-home\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-home\") pod \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") "
Oct 30 19:17:33.168752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168580    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-dshm\") pod \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") "
Oct 30 19:17:33.168752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168622    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tls-certs\") pod \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") "
Oct 30 19:17:33.168752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168683    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-model-cache\") pod \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") "
Oct 30 19:17:33.168752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168713    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-kube-api-access-7d74j\" (UniqueName: \"kubernetes.io/projected/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-kube-api-access-7d74j\") pod \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") "
Oct 30 19:17:33.168969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168803    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-dshm\") pod \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") "
Oct 30 19:17:33.168969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168804    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-home" pod "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" (UID: "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:33.168969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168832    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tmp-dir\") pod \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") "
Oct 30 19:17:33.168969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168880    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-kube-api-access-fzrmv\" (UniqueName: \"kubernetes.io/projected/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-kube-api-access-fzrmv\") pod \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") "
Oct 30 19:17:33.168969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168945    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-model-cache\") pod \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") "
Oct 30 19:17:33.169280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.168988    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tls-certs\" (UniqueName: \"kubernetes.io/secret/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tls-certs\") pod \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") "
Oct 30 19:17:33.169280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.169024    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tmp-dir\") pod \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\" (UID: \"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e\") "
Oct 30 19:17:33.169280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.169050    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-home\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-home\") pod \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\" (UID: \"60ba4ae1-8289-4d1a-8e25-88c5fc57494b\") "
Oct 30 19:17:33.175206 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.169636    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.175206 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.170003    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-home" pod "60ba4ae1-8289-4d1a-8e25-88c5fc57494b" (UID: "60ba4ae1-8289-4d1a-8e25-88c5fc57494b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:33.175206 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.170111    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-model-cache" pod "60ba4ae1-8289-4d1a-8e25-88c5fc57494b" (UID: "60ba4ae1-8289-4d1a-8e25-88c5fc57494b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:33.178112 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.177606    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tmp-dir" pod "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" (UID: "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:33.178112 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.177875    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tmp-dir" pod "60ba4ae1-8289-4d1a-8e25-88c5fc57494b" (UID: "60ba4ae1-8289-4d1a-8e25-88c5fc57494b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:33.178112 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.178053    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-model-cache" pod "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" (UID: "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:33.180091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.180037    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tls-certs" pod "60ba4ae1-8289-4d1a-8e25-88c5fc57494b" (UID: "60ba4ae1-8289-4d1a-8e25-88c5fc57494b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:33.181637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.181611    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-dshm" pod "60ba4ae1-8289-4d1a-8e25-88c5fc57494b" (UID: "60ba4ae1-8289-4d1a-8e25-88c5fc57494b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:33.181763 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.181722    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tls-certs" pod "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" (UID: "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:33.182738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.182714    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-kube-api-access-7d74j" pod "60ba4ae1-8289-4d1a-8e25-88c5fc57494b" (UID: "60ba4ae1-8289-4d1a-8e25-88c5fc57494b"). InnerVolumeSpecName "kube-api-access-7d74j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:33.182843 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.182794    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-dshm" pod "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" (UID: "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:33.183232 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.183205    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-kube-api-access-fzrmv" pod "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" (UID: "4b3b069b-5a17-4b68-afcb-4c444c3e3f0e"). InnerVolumeSpecName "kube-api-access-fzrmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:33.233516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.228831    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078a0cc5-96c9-49b4-bd07-91cc514a556e" path="/var/lib/kubelet/pods/078a0cc5-96c9-49b4-bd07-91cc514a556e/volumes"
Oct 30 19:17:33.271053 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271013    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzrmv\" (UniqueName: \"kubernetes.io/projected/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-kube-api-access-fzrmv\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.271053 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271056    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.271254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271072    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.271254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271086    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.271254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271099    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.271254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271111    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.271254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271124    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.271254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271137    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.271254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271152    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d74j\" (UniqueName: \"kubernetes.io/projected/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-kube-api-access-7d74j\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.271254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271165    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.271254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.271180    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ba4ae1-8289-4d1a-8e25-88c5fc57494b-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:33.346999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.346901    4267 generic.go:356] "Generic (PLEG): container finished" podID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerID="8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d" exitCode=0
Oct 30 19:17:33.347467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.347087    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" event={"ID":"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e","Type":"ContainerDied","Data":"8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d"}
Oct 30 19:17:33.347467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.347131    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g" event={"ID":"4b3b069b-5a17-4b68-afcb-4c444c3e3f0e","Type":"ContainerDied","Data":"8cecca216306b5085a855f815c34fb4c2b0b35b8708d4bd1e28cfcb9b053c0b9"}
Oct 30 19:17:33.347467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.347155    4267 scope.go:122] "RemoveContainer" containerID="8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d"
Oct 30 19:17:33.347467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.347300    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"
Oct 30 19:17:33.353127 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.353098    4267 generic.go:356] "Generic (PLEG): container finished" podID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerID="c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9" exitCode=0
Oct 30 19:17:33.353369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.353165    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" event={"ID":"60ba4ae1-8289-4d1a-8e25-88c5fc57494b","Type":"ContainerDied","Data":"c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9"}
Oct 30 19:17:33.353369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.353197    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc" event={"ID":"60ba4ae1-8289-4d1a-8e25-88c5fc57494b","Type":"ContainerDied","Data":"bc6fabd1c91414156ead8ae00b6357f18f536388a6a295d1d6ec26b73807958f"}
Oct 30 19:17:33.353369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.353262    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"
Oct 30 19:17:33.368702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.368672    4267 scope.go:122] "RemoveContainer" containerID="8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d"
Oct 30 19:17:33.369051 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:33.369030    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d\": container with ID starting with 8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d not found: ID does not exist" containerID="8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d"
Oct 30 19:17:33.369139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.369062    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d"} err="failed to get container status \"8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d\": rpc error: code = NotFound desc = could not find container \"8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d\": container with ID starting with 8bf18faf98788c6b0b89b3da97b89f9cb3a82c4a8feb702e55a58a3d368da05d not found: ID does not exist"
Oct 30 19:17:33.369139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.369080    4267 scope.go:122] "RemoveContainer" containerID="c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9"
Oct 30 19:17:33.381694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.381672    4267 scope.go:122] "RemoveContainer" containerID="c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9"
Oct 30 19:17:33.382018 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:33.381996    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9\": container with ID starting with c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9 not found: ID does not exist" containerID="c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9"
Oct 30 19:17:33.382082 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.382026    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9"} err="failed to get container status \"c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9\": rpc error: code = NotFound desc = could not find container \"c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9\": container with ID starting with c04a04731864908927dba1a0cd6ffd6649e8ac7c19ea869067b62b152db6fae9 not found: ID does not exist"
Oct 30 19:17:33.516015 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.515980    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"]
Oct 30 19:17:33.523377 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.523344    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/premium-w6-41db7c-kserve-56b677d849-2rr4g"]
Oct 30 19:17:33.632327 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.632238    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"]
Oct 30 19:17:33.638354 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:33.638317    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-41db7c/simulator-w6-41db7c-kserve-7cb895548d-k24kc"]
Oct 30 19:17:34.004412 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.004368    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r" podUID="c747f06b-e604-4aa9-a122-5a969cec8306" containerName="payload-processing" containerID="cri-o://d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089" gracePeriod=25
Oct 30 19:17:34.114631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.114580    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n" podUID="c9e6a4ee-ad97-40b0-a608-82756a285d9d" containerName="payload-pre-processing" containerID="cri-o://39aadef5027ef5d5e2fe9b259c7af99759e265eede0fe1d76e2bd3fd7ea00ae0" gracePeriod=25
Oct 30 19:17:34.331523 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.329299    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"]
Oct 30 19:17:34.335103 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.335051    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24e88a-kq9vh"]
Oct 30 19:17:34.355866 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.355254    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:17:34.368355 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.366038    4267 generic.go:356] "Generic (PLEG): container finished" podID="c9e6a4ee-ad97-40b0-a608-82756a285d9d" containerID="39aadef5027ef5d5e2fe9b259c7af99759e265eede0fe1d76e2bd3fd7ea00ae0" exitCode=0
Oct 30 19:17:34.368355 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.366112    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n" event={"ID":"c9e6a4ee-ad97-40b0-a608-82756a285d9d","Type":"ContainerDied","Data":"39aadef5027ef5d5e2fe9b259c7af99759e265eede0fe1d76e2bd3fd7ea00ae0"}
Oct 30 19:17:34.369528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.368641    4267 generic.go:356] "Generic (PLEG): container finished" podID="c747f06b-e604-4aa9-a122-5a969cec8306" containerID="d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089" exitCode=0
Oct 30 19:17:34.369528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.368665    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r" event={"ID":"c747f06b-e604-4aa9-a122-5a969cec8306","Type":"ContainerDied","Data":"d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089"}
Oct 30 19:17:34.369528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.368703    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r" event={"ID":"c747f06b-e604-4aa9-a122-5a969cec8306","Type":"ContainerDied","Data":"f06d63b612a5521277d7c19d9ee9ec3f1f41a87067597ad7b2c5468a1e354125"}
Oct 30 19:17:34.369528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.368712    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"
Oct 30 19:17:34.369528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.368725    4267 scope.go:122] "RemoveContainer" containerID="d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089"
Oct 30 19:17:34.383415 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.383393    4267 scope.go:122] "RemoveContainer" containerID="d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089"
Oct 30 19:17:34.384040 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:34.383957    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089\": container with ID starting with d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089 not found: ID does not exist" containerID="d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089"
Oct 30 19:17:34.384040 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.383995    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089"} err="failed to get container status \"d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089\": rpc error: code = NotFound desc = could not find container \"d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089\": container with ID starting with d7c1460055a1d8be04b41d892c0f7381112185a5d5f8e1663f848668b4346089 not found: ID does not exist"
Oct 30 19:17:34.385474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.385372    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c747f06b-e604-4aa9-a122-5a969cec8306-kube-api-access-p96x2\" (UniqueName: \"kubernetes.io/projected/c747f06b-e604-4aa9-a122-5a969cec8306-kube-api-access-p96x2\") pod \"c747f06b-e604-4aa9-a122-5a969cec8306\" (UID: \"c747f06b-e604-4aa9-a122-5a969cec8306\") "
Oct 30 19:17:34.387008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.385886    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c747f06b-e604-4aa9-a122-5a969cec8306-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c747f06b-e604-4aa9-a122-5a969cec8306-plugins-config-volume\") pod \"c747f06b-e604-4aa9-a122-5a969cec8306\" (UID: \"c747f06b-e604-4aa9-a122-5a969cec8306\") "
Oct 30 19:17:34.387008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.386842    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c747f06b-e604-4aa9-a122-5a969cec8306-plugins-config-volume" pod "c747f06b-e604-4aa9-a122-5a969cec8306" (UID: "c747f06b-e604-4aa9-a122-5a969cec8306"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:34.389211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.389181    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c747f06b-e604-4aa9-a122-5a969cec8306-kube-api-access-p96x2" pod "c747f06b-e604-4aa9-a122-5a969cec8306" (UID: "c747f06b-e604-4aa9-a122-5a969cec8306"). InnerVolumeSpecName "kube-api-access-p96x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:34.488178 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.488134    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p96x2\" (UniqueName: \"kubernetes.io/projected/c747f06b-e604-4aa9-a122-5a969cec8306-kube-api-access-p96x2\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.488178 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.488180    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c747f06b-e604-4aa9-a122-5a969cec8306-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.500324 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.500300    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:17:34.588972 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.588935    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e6a4ee-ad97-40b0-a608-82756a285d9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e6a4ee-ad97-40b0-a608-82756a285d9d-plugins-config-volume\") pod \"c9e6a4ee-ad97-40b0-a608-82756a285d9d\" (UID: \"c9e6a4ee-ad97-40b0-a608-82756a285d9d\") "
Oct 30 19:17:34.589167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.589137    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e6a4ee-ad97-40b0-a608-82756a285d9d-kube-api-access-8m247\" (UniqueName: \"kubernetes.io/projected/c9e6a4ee-ad97-40b0-a608-82756a285d9d-kube-api-access-8m247\") pod \"c9e6a4ee-ad97-40b0-a608-82756a285d9d\" (UID: \"c9e6a4ee-ad97-40b0-a608-82756a285d9d\") "
Oct 30 19:17:34.589390 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.589365    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e6a4ee-ad97-40b0-a608-82756a285d9d-plugins-config-volume" pod "c9e6a4ee-ad97-40b0-a608-82756a285d9d" (UID: "c9e6a4ee-ad97-40b0-a608-82756a285d9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:34.589633 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.589616    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e6a4ee-ad97-40b0-a608-82756a285d9d-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.591181 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.591160    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e6a4ee-ad97-40b0-a608-82756a285d9d-kube-api-access-8m247" pod "c9e6a4ee-ad97-40b0-a608-82756a285d9d" (UID: "c9e6a4ee-ad97-40b0-a608-82756a285d9d"). InnerVolumeSpecName "kube-api-access-8m247". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:34.687560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.687480    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:17:34.691113 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.691089    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m247\" (UniqueName: \"kubernetes.io/projected/c9e6a4ee-ad97-40b0-a608-82756a285d9d-kube-api-access-8m247\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.792069 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792033    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-data\") pod \"1e17b05e-8313-497d-a3d2-abfd08e4db51\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") "
Oct 30 19:17:34.792265 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792111    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-credential-socket\") pod \"1e17b05e-8313-497d-a3d2-abfd08e4db51\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") "
Oct 30 19:17:34.792265 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792169    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-token\" (UniqueName: \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-token\") pod \"1e17b05e-8313-497d-a3d2-abfd08e4db51\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") "
Oct 30 19:17:34.792265 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792237    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-kube-api-access-52flr\" (UniqueName: \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-kube-api-access-52flr\") pod \"1e17b05e-8313-497d-a3d2-abfd08e4db51\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") "
Oct 30 19:17:34.792452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792277    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-certs\") pod \"1e17b05e-8313-497d-a3d2-abfd08e4db51\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") "
Oct 30 19:17:34.792452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792300    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-data" pod "1e17b05e-8313-497d-a3d2-abfd08e4db51" (UID: "1e17b05e-8313-497d-a3d2-abfd08e4db51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:34.792452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792367    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-podinfo\") pod \"1e17b05e-8313-497d-a3d2-abfd08e4db51\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") "
Oct 30 19:17:34.792452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792418    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-envoy\") pod \"1e17b05e-8313-497d-a3d2-abfd08e4db51\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") "
Oct 30 19:17:34.792702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792520    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-socket\") pod \"1e17b05e-8313-497d-a3d2-abfd08e4db51\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") "
Oct 30 19:17:34.792702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792557    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e17b05e-8313-497d-a3d2-abfd08e4db51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e17b05e-8313-497d-a3d2-abfd08e4db51-istiod-ca-cert\") pod \"1e17b05e-8313-497d-a3d2-abfd08e4db51\" (UID: \"1e17b05e-8313-497d-a3d2-abfd08e4db51\") "
Oct 30 19:17:34.792916 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.792819    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-credential-socket" pod "1e17b05e-8313-497d-a3d2-abfd08e4db51" (UID: "1e17b05e-8313-497d-a3d2-abfd08e4db51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:34.793384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.793070    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-certs" pod "1e17b05e-8313-497d-a3d2-abfd08e4db51" (UID: "1e17b05e-8313-497d-a3d2-abfd08e4db51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:34.793384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.793280    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-socket" pod "1e17b05e-8313-497d-a3d2-abfd08e4db51" (UID: "1e17b05e-8313-497d-a3d2-abfd08e4db51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:34.793384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.793338    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e17b05e-8313-497d-a3d2-abfd08e4db51-istiod-ca-cert" pod "1e17b05e-8313-497d-a3d2-abfd08e4db51" (UID: "1e17b05e-8313-497d-a3d2-abfd08e4db51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:34.793570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.793458    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.793570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.793477    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.793570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.793519    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e17b05e-8313-497d-a3d2-abfd08e4db51-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.793570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.793535    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.794740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.794715    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-token" pod "1e17b05e-8313-497d-a3d2-abfd08e4db51" (UID: "1e17b05e-8313-497d-a3d2-abfd08e4db51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:34.795236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.795216    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-podinfo" pod "1e17b05e-8313-497d-a3d2-abfd08e4db51" (UID: "1e17b05e-8313-497d-a3d2-abfd08e4db51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:34.795750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.795716    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-envoy" pod "1e17b05e-8313-497d-a3d2-abfd08e4db51" (UID: "1e17b05e-8313-497d-a3d2-abfd08e4db51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:34.795750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.795734    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-kube-api-access-52flr" pod "1e17b05e-8313-497d-a3d2-abfd08e4db51" (UID: "1e17b05e-8313-497d-a3d2-abfd08e4db51"). InnerVolumeSpecName "kube-api-access-52flr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:34.798176 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.798151    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"]
Oct 30 19:17:34.801375 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.801355    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-903a82-7b5fbff5bc-28z9r"]
Oct 30 19:17:34.894578 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.894547    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.894578 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.894575    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.894578 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.894585    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52flr\" (UniqueName: \"kubernetes.io/projected/1e17b05e-8313-497d-a3d2-abfd08e4db51-kube-api-access-52flr\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.894811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.894594    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.894811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.894605    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e17b05e-8313-497d-a3d2-abfd08e4db51-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:34.909565 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.909525    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"]
Oct 30 19:17:34.918113 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:34.918078    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"]
Oct 30 19:17:35.223845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.223808    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" path="/var/lib/kubelet/pods/4b3b069b-5a17-4b68-afcb-4c444c3e3f0e/volumes"
Oct 30 19:17:35.224260 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.224245    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" path="/var/lib/kubelet/pods/60ba4ae1-8289-4d1a-8e25-88c5fc57494b/volumes"
Oct 30 19:17:35.224653 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.224637    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c747f06b-e604-4aa9-a122-5a969cec8306" path="/var/lib/kubelet/pods/c747f06b-e604-4aa9-a122-5a969cec8306/volumes"
Oct 30 19:17:35.225341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.225325    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04bcaa3-94a4-43ed-b307-e7cb51c67b45" path="/var/lib/kubelet/pods/f04bcaa3-94a4-43ed-b307-e7cb51c67b45/volumes"
Oct 30 19:17:35.377105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.377077    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"
Oct 30 19:17:35.377598 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.377068    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n" event={"ID":"c9e6a4ee-ad97-40b0-a608-82756a285d9d","Type":"ContainerDied","Data":"c3b516ec7e02ef8ce146cd702136b1f5f2ff9817cecbcabd41a8877664e57065"}
Oct 30 19:17:35.377598 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.377226    4267 scope.go:122] "RemoveContainer" containerID="39aadef5027ef5d5e2fe9b259c7af99759e265eede0fe1d76e2bd3fd7ea00ae0"
Oct 30 19:17:35.378655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.378635    4267 generic.go:356] "Generic (PLEG): container finished" podID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerID="92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606" exitCode=0
Oct 30 19:17:35.378748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.378682    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"
Oct 30 19:17:35.378748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.378724    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" event={"ID":"1e17b05e-8313-497d-a3d2-abfd08e4db51","Type":"ContainerDied","Data":"92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606"}
Oct 30 19:17:35.378850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.378779    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q" event={"ID":"1e17b05e-8313-497d-a3d2-abfd08e4db51","Type":"ContainerDied","Data":"b080f63a8369a63632ad286f588eb9486c496aaa47c7d53c10ff592751568b58"}
Oct 30 19:17:35.387816 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.387800    4267 scope.go:122] "RemoveContainer" containerID="92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606"
Oct 30 19:17:35.397001 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.396985    4267 scope.go:122] "RemoveContainer" containerID="92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606"
Oct 30 19:17:35.397285 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:35.397259    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606\": container with ID starting with 92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606 not found: ID does not exist" containerID="92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606"
Oct 30 19:17:35.397326 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.397292    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606"} err="failed to get container status \"92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606\": rpc error: code = NotFound desc = could not find container \"92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606\": container with ID starting with 92446f6d4c87aebfab679244fbaf3f41e16a079b51daaf614b7ba38b08c52606 not found: ID does not exist"
Oct 30 19:17:35.545948 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.545864    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"]
Oct 30 19:17:35.551488 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.551462    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-903a82-5cf8bdcb8c-npm2n"]
Oct 30 19:17:35.661877 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.661837    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"]
Oct 30 19:17:35.665917 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:35.665893    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-87ed12-openshift-default-7ddcd7cf97-xbn5q"]
Oct 30 19:17:37.221428 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:37.221396    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" path="/var/lib/kubelet/pods/1e17b05e-8313-497d-a3d2-abfd08e4db51/volumes"
Oct 30 19:17:37.221934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:37.221851    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e6a4ee-ad97-40b0-a608-82756a285d9d" path="/var/lib/kubelet/pods/c9e6a4ee-ad97-40b0-a608-82756a285d9d/volumes"
Oct 30 19:17:38.794433 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:38.794401    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj" podUID="81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:41.058771 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:41.058741    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"]
Oct 30 19:17:41.059347 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:41.058933    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy" containerID="cri-o://4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72" gracePeriod=30
Oct 30 19:17:41.062198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:41.062171    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"]
Oct 30 19:17:42.027109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.027076    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"]
Oct 30 19:17:42.027882 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.027860    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c747f06b-e604-4aa9-a122-5a969cec8306" containerName="payload-processing"
Oct 30 19:17:42.027953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.027885    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c747f06b-e604-4aa9-a122-5a969cec8306" containerName="payload-processing"
Oct 30 19:17:42.027953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.027924    4267 cpu_manager.go:414] "containerMap: removing container" podUID="078a0cc5-96c9-49b4-bd07-91cc514a556e" containerName="maas-api"
Oct 30 19:17:42.027953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.027933    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078a0cc5-96c9-49b4-bd07-91cc514a556e" containerName="maas-api"
Oct 30 19:17:42.027953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.027946    4267 cpu_manager.go:414] "containerMap: removing container" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main"
Oct 30 19:17:42.028117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.027953    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main"
Oct 30 19:17:42.028117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.027971    4267 cpu_manager.go:414] "containerMap: removing container" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main"
Oct 30 19:17:42.028117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.027978    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main"
Oct 30 19:17:42.028117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028001    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c9e6a4ee-ad97-40b0-a608-82756a285d9d" containerName="payload-pre-processing"
Oct 30 19:17:42.028117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028009    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e6a4ee-ad97-40b0-a608-82756a285d9d" containerName="payload-pre-processing"
Oct 30 19:17:42.028117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028032    4267 cpu_manager.go:414] "containerMap: removing container" podUID="92408e1f-b0f2-4d10-9899-71656d977ed6" containerName="revoke-keys"
Oct 30 19:17:42.028117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028040    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92408e1f-b0f2-4d10-9899-71656d977ed6" containerName="revoke-keys"
Oct 30 19:17:42.028117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028051    4267 cpu_manager.go:414] "containerMap: removing container" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerName="istio-proxy"
Oct 30 19:17:42.028117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028058    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerName="istio-proxy"
Oct 30 19:17:42.028401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028179    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ba4ae1-8289-4d1a-8e25-88c5fc57494b" containerName="main"
Oct 30 19:17:42.028401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028192    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3b069b-5a17-4b68-afcb-4c444c3e3f0e" containerName="main"
Oct 30 19:17:42.028401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028205    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e6a4ee-ad97-40b0-a608-82756a285d9d" containerName="payload-pre-processing"
Oct 30 19:17:42.028401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028218    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="92408e1f-b0f2-4d10-9899-71656d977ed6" containerName="revoke-keys"
Oct 30 19:17:42.028401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028226    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e17b05e-8313-497d-a3d2-abfd08e4db51" containerName="istio-proxy"
Oct 30 19:17:42.028401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028236    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e4ed6e-a888-486b-9a6a-329bd8e2d5e5" containerName="istio-proxy"
Oct 30 19:17:42.028401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028245    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c747f06b-e604-4aa9-a122-5a969cec8306" containerName="payload-processing"
Oct 30 19:17:42.028401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.028256    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="078a0cc5-96c9-49b4-bd07-91cc514a556e" containerName="maas-api"
Oct 30 19:17:42.033429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.033410    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.085929 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.085874    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:17:42.086904 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.086880    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:42.088822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.088788    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8dln\" (UniqueName: \"kubernetes.io/projected/0ef3a653-91ad-4038-af04-eba703cc0644-kube-api-access-x8dln\") pod \"maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.088948 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.088840    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.088948 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.088865    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.089550 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.089525    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"]
Oct 30 19:17:42.190251 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.190212    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8dln\" (UniqueName: \"kubernetes.io/projected/0ef3a653-91ad-4038-af04-eba703cc0644-kube-api-access-x8dln\") pod \"maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.190449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.190268    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.190449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.190299    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.191248 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.191216    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.192784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.192757    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.246780 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.246749    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8dln\" (UniqueName: \"kubernetes.io/projected/0ef3a653-91ad-4038-af04-eba703cc0644-kube-api-access-x8dln\") pod \"maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.348853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.348754    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:42.857298 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:17:42.857260    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef3a653_91ad_4038_af04_eba703cc0644.slice/crio-7983d183d356ba2f263c897e4eefc7fe63b04bdfb3fa444decf105811a78bc4a WatchSource:0}: Error finding container 7983d183d356ba2f263c897e4eefc7fe63b04bdfb3fa444decf105811a78bc4a: Status 404 returned error can't find the container with id 7983d183d356ba2f263c897e4eefc7fe63b04bdfb3fa444decf105811a78bc4a
Oct 30 19:17:42.857298 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:42.857283    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"]
Oct 30 19:17:43.488389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:43.488344    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44" event={"ID":"0ef3a653-91ad-4038-af04-eba703cc0644","Type":"ContainerStarted","Data":"39ef61a91a34457d5cac782936805f14d956b34c3777b3eee6898365110400aa"}
Oct 30 19:17:43.488985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:43.488419    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44" event={"ID":"0ef3a653-91ad-4038-af04-eba703cc0644","Type":"ContainerStarted","Data":"7983d183d356ba2f263c897e4eefc7fe63b04bdfb3fa444decf105811a78bc4a"}
Oct 30 19:17:43.604052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:43.603993    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44" podStartSLOduration=2.603979863 podStartE2EDuration="2.603979863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:43.600967982 +0000 UTC m=+2354.488885073" watchObservedRunningTime="2026-10-30 19:17:43.603979863 +0000 UTC m=+2354.491896951"
Oct 30 19:17:44.395506 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:44.395456    4267 patch_prober.go:29] interesting pod/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:17:44.395694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:44.395545    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:17:44.497134 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:44.497095    4267 generic.go:356] "Generic (PLEG): container finished" podID="0ef3a653-91ad-4038-af04-eba703cc0644" containerID="39ef61a91a34457d5cac782936805f14d956b34c3777b3eee6898365110400aa" exitCode=0
Oct 30 19:17:44.497587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:44.497188    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44" event={"ID":"0ef3a653-91ad-4038-af04-eba703cc0644","Type":"ContainerDied","Data":"39ef61a91a34457d5cac782936805f14d956b34c3777b3eee6898365110400aa"}
Oct 30 19:17:45.672572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:45.672545    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:45.750685 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:45.750654    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-xks-ca\") pod \"0ef3a653-91ad-4038-af04-eba703cc0644\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") "
Oct 30 19:17:45.750851 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:45.750738    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-openshift-ca\") pod \"0ef3a653-91ad-4038-af04-eba703cc0644\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") "
Oct 30 19:17:45.750851 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:45.750787    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef3a653-91ad-4038-af04-eba703cc0644-kube-api-access-x8dln\" (UniqueName: \"kubernetes.io/projected/0ef3a653-91ad-4038-af04-eba703cc0644-kube-api-access-x8dln\") pod \"0ef3a653-91ad-4038-af04-eba703cc0644\" (UID: \"0ef3a653-91ad-4038-af04-eba703cc0644\") "
Oct 30 19:17:45.751143 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:45.751117    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-openshift-ca" pod "0ef3a653-91ad-4038-af04-eba703cc0644" (UID: "0ef3a653-91ad-4038-af04-eba703cc0644"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:45.751369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:45.751346    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:45.752913 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:45.752882    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-xks-ca" pod "0ef3a653-91ad-4038-af04-eba703cc0644" (UID: "0ef3a653-91ad-4038-af04-eba703cc0644"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:45.753048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:45.753021    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef3a653-91ad-4038-af04-eba703cc0644-kube-api-access-x8dln" pod "0ef3a653-91ad-4038-af04-eba703cc0644" (UID: "0ef3a653-91ad-4038-af04-eba703cc0644"). InnerVolumeSpecName "kube-api-access-x8dln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:45.852953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:45.852919    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef3a653-91ad-4038-af04-eba703cc0644-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:45.852953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:45.852948    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8dln\" (UniqueName: \"kubernetes.io/projected/0ef3a653-91ad-4038-af04-eba703cc0644-kube-api-access-x8dln\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.063012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.062923    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r" podUID="4547d45a-763d-485c-9c2f-9433013e97af" containerName="payload-pre-processing" containerID="cri-o://d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160" gracePeriod=25
Oct 30 19:17:46.327703 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.327677    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:17:46.358428 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358400    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-certs\") pod \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") "
Oct 30 19:17:46.358614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358434    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-envoy\") pod \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") "
Oct 30 19:17:46.358614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358458    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istiod-ca-cert\") pod \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") "
Oct 30 19:17:46.358614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358525    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-podinfo\") pod \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") "
Oct 30 19:17:46.358806 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358689    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-token\" (UniqueName: \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-token\") pod \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") "
Oct 30 19:17:46.358806 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358719    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-certs" pod "60ce3cd7-8197-4758-bfc1-cff2f15d12ae" (UID: "60ce3cd7-8197-4758-bfc1-cff2f15d12ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:46.358806 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358733    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-credential-socket\") pod \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") "
Oct 30 19:17:46.358806 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358804    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-kube-api-access-48mlx\" (UniqueName: \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-kube-api-access-48mlx\") pod \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") "
Oct 30 19:17:46.359023 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358858    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-data\") pod \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") "
Oct 30 19:17:46.359023 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358888    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-socket\") pod \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\" (UID: \"60ce3cd7-8197-4758-bfc1-cff2f15d12ae\") "
Oct 30 19:17:46.359023 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.358910    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istiod-ca-cert" pod "60ce3cd7-8197-4758-bfc1-cff2f15d12ae" (UID: "60ce3cd7-8197-4758-bfc1-cff2f15d12ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:46.359179 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.359124    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-data" pod "60ce3cd7-8197-4758-bfc1-cff2f15d12ae" (UID: "60ce3cd7-8197-4758-bfc1-cff2f15d12ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:46.359333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.359298    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-credential-socket" pod "60ce3cd7-8197-4758-bfc1-cff2f15d12ae" (UID: "60ce3cd7-8197-4758-bfc1-cff2f15d12ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:46.359333 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.359312    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-socket" pod "60ce3cd7-8197-4758-bfc1-cff2f15d12ae" (UID: "60ce3cd7-8197-4758-bfc1-cff2f15d12ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:46.359487 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.359468    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.359553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.359526    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.359553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.359545    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.359622 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.359560    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.359622 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.359574    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.360832 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.360810    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-token" pod "60ce3cd7-8197-4758-bfc1-cff2f15d12ae" (UID: "60ce3cd7-8197-4758-bfc1-cff2f15d12ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:46.360928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.360869    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-kube-api-access-48mlx" pod "60ce3cd7-8197-4758-bfc1-cff2f15d12ae" (UID: "60ce3cd7-8197-4758-bfc1-cff2f15d12ae"). InnerVolumeSpecName "kube-api-access-48mlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:46.361118 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.361099    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-podinfo" pod "60ce3cd7-8197-4758-bfc1-cff2f15d12ae" (UID: "60ce3cd7-8197-4758-bfc1-cff2f15d12ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:46.361194 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.361096    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-envoy" pod "60ce3cd7-8197-4758-bfc1-cff2f15d12ae" (UID: "60ce3cd7-8197-4758-bfc1-cff2f15d12ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:46.461271 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.461237    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.461271 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.461265    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.461271 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.461275    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.461533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.461284    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48mlx\" (UniqueName: \"kubernetes.io/projected/60ce3cd7-8197-4758-bfc1-cff2f15d12ae-kube-api-access-48mlx\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.517048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.517014    4267 generic.go:356] "Generic (PLEG): container finished" podID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerID="4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72" exitCode=0
Oct 30 19:17:46.517236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.517087    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"
Oct 30 19:17:46.517236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.517100    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" event={"ID":"60ce3cd7-8197-4758-bfc1-cff2f15d12ae","Type":"ContainerDied","Data":"4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72"}
Oct 30 19:17:46.517236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.517155    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk" event={"ID":"60ce3cd7-8197-4758-bfc1-cff2f15d12ae","Type":"ContainerDied","Data":"ab6e2e80d7edd4d93f09284008dde5dc78335e761214692de4b0ac42feb700d4"}
Oct 30 19:17:46.517236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.517183    4267 scope.go:122] "RemoveContainer" containerID="4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72"
Oct 30 19:17:46.519058 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.519034    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44" event={"ID":"0ef3a653-91ad-4038-af04-eba703cc0644","Type":"ContainerDied","Data":"7983d183d356ba2f263c897e4eefc7fe63b04bdfb3fa444decf105811a78bc4a"}
Oct 30 19:17:46.519187 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.519079    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7983d183d356ba2f263c897e4eefc7fe63b04bdfb3fa444decf105811a78bc4a"
Oct 30 19:17:46.519187 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.519112    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"
Oct 30 19:17:46.541070 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.541042    4267 scope.go:122] "RemoveContainer" containerID="aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75"
Oct 30 19:17:46.553658 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.553627    4267 scope.go:122] "RemoveContainer" containerID="4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72"
Oct 30 19:17:46.553970 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:46.553943    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72\": container with ID starting with 4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72 not found: ID does not exist" containerID="4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72"
Oct 30 19:17:46.554038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.553984    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72"} err="failed to get container status \"4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72\": rpc error: code = NotFound desc = could not find container \"4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72\": container with ID starting with 4fdaa3637e2c841826c5832788a3278ce2217281e4ae995ad16b24a3b69c8c72 not found: ID does not exist"
Oct 30 19:17:46.554038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.554011    4267 scope.go:122] "RemoveContainer" containerID="aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75"
Oct 30 19:17:46.554255 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:46.554237    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75\": container with ID starting with aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75 not found: ID does not exist" containerID="aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75"
Oct 30 19:17:46.554304 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.554262    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75"} err="failed to get container status \"aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75\": rpc error: code = NotFound desc = could not find container \"aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75\": container with ID starting with aad2e241b63ada00c61a36d08f8ea0f980187607f86a378703a685c054ef8e75 not found: ID does not exist"
Oct 30 19:17:46.662192 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.662159    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"]
Oct 30 19:17:46.686105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.686074    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"]
Oct 30 19:17:46.693778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:46.693740    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-24e88a-openshift-default-7b779f6c8c-lrblk"]
Oct 30 19:17:47.221359 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:47.221327    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" path="/var/lib/kubelet/pods/60ce3cd7-8197-4758-bfc1-cff2f15d12ae/volumes"
Oct 30 19:17:48.793725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.793688    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj" podUID="81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:48.823066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.823032    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"]
Oct 30 19:17:48.823598 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.823576    4267 cpu_manager.go:414] "containerMap: removing container" podUID="0ef3a653-91ad-4038-af04-eba703cc0644" containerName="revoke-keys"
Oct 30 19:17:48.823598 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.823594    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef3a653-91ad-4038-af04-eba703cc0644" containerName="revoke-keys"
Oct 30 19:17:48.823808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.823622    4267 cpu_manager.go:414] "containerMap: removing container" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy"
Oct 30 19:17:48.823808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.823628    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy"
Oct 30 19:17:48.823808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.823645    4267 cpu_manager.go:414] "containerMap: removing container" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy"
Oct 30 19:17:48.823808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.823652    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy"
Oct 30 19:17:48.823808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.823741    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy"
Oct 30 19:17:48.823808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.823750    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ce3cd7-8197-4758-bfc1-cff2f15d12ae" containerName="istio-proxy"
Oct 30 19:17:48.823808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.823759    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef3a653-91ad-4038-af04-eba703cc0644" containerName="revoke-keys"
Oct 30 19:17:48.828841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.828816    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:48.889059 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.889024    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grzb\" (UniqueName: \"kubernetes.io/projected/c1162287-d097-4762-a3b6-28e28c7910e1-kube-api-access-4grzb\") pod \"maas-api-revoke-sub-35a894cb-chmzx\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:48.889249 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.889209    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35a894cb-chmzx\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:48.889369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.889348    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35a894cb-chmzx\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:48.903803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.903770    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"]
Oct 30 19:17:48.907763 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.907740    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"]
Oct 30 19:17:48.907902 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.907842    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:48.932785 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.932756    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:48.932971 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.932836    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:17:48.954036 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.954006    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"]
Oct 30 19:17:48.990741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.990704    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35a894cb-chmzx\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:48.990954 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.990777    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa2cfd3f-9w7nb\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:48.990954 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.990841    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grzb\" (UniqueName: \"kubernetes.io/projected/c1162287-d097-4762-a3b6-28e28c7910e1-kube-api-access-4grzb\") pod \"maas-api-revoke-sub-35a894cb-chmzx\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:48.991082 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.991048    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa2cfd3f-9w7nb\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:48.991140 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.991107    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35a894cb-chmzx\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:48.991257 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.991237    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvjj\" (UniqueName: \"kubernetes.io/projected/662447d4-f16d-458e-ba57-ce7cc87bcc8c-kube-api-access-2jvjj\") pod \"maas-api-revoke-sub-fa2cfd3f-9w7nb\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:48.991960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.991932    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35a894cb-chmzx\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:48.993275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:48.993253    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35a894cb-chmzx\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:49.048545 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.048426    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grzb\" (UniqueName: \"kubernetes.io/projected/c1162287-d097-4762-a3b6-28e28c7910e1-kube-api-access-4grzb\") pod \"maas-api-revoke-sub-35a894cb-chmzx\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:49.092886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.092849    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa2cfd3f-9w7nb\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:49.093075 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.092933    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvjj\" (UniqueName: \"kubernetes.io/projected/662447d4-f16d-458e-ba57-ce7cc87bcc8c-kube-api-access-2jvjj\") pod \"maas-api-revoke-sub-fa2cfd3f-9w7nb\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:49.093075 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.092989    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa2cfd3f-9w7nb\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:49.093738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.093717    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa2cfd3f-9w7nb\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:49.095226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.095203    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa2cfd3f-9w7nb\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:49.145464 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.145425    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:49.150809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.150780    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvjj\" (UniqueName: \"kubernetes.io/projected/662447d4-f16d-458e-ba57-ce7cc87bcc8c-kube-api-access-2jvjj\") pod \"maas-api-revoke-sub-fa2cfd3f-9w7nb\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:49.227109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.226711    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:49.549443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.549412    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"]
Oct 30 19:17:49.550127 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.550103    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx" event={"ID":"c1162287-d097-4762-a3b6-28e28c7910e1","Type":"ContainerStarted","Data":"432f7e70563c3da0848f7945b8a24c9480ff828694ac7643010b67b3635ebc05"}
Oct 30 19:17:49.552270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.552211    4267 generic.go:356] "Generic (PLEG): container finished" podID="81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa" containerID="b06dee0f36c748139250a94141aa7a998af464415d329827dfa0b8fc2b6d87ee" exitCode=0
Oct 30 19:17:49.552270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.552236    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj" event={"ID":"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa","Type":"ContainerDied","Data":"b06dee0f36c748139250a94141aa7a998af464415d329827dfa0b8fc2b6d87ee"}
Oct 30 19:17:49.612916 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.612886    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 19:17:49.640226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.640185    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"]
Oct 30 19:17:49.700601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.700568    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-plugins-config-volume\") pod \"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa\" (UID: \"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa\") "
Oct 30 19:17:49.700801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.700719    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-kube-api-access-bn8dx\" (UniqueName: \"kubernetes.io/projected/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-kube-api-access-bn8dx\") pod \"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa\" (UID: \"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa\") "
Oct 30 19:17:49.700994 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.700969    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-plugins-config-volume" pod "81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa" (UID: "81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:49.701169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.701151    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:49.702761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.702740    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-kube-api-access-bn8dx" pod "81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa" (UID: "81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa"). InnerVolumeSpecName "kube-api-access-bn8dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:49.829399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.804185    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn8dx\" (UniqueName: \"kubernetes.io/projected/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa-kube-api-access-bn8dx\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:49.946842 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.946776    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" containerID="cri-o://021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f" gracePeriod=45
Oct 30 19:17:49.952709 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:49.952653    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" containerID="cri-o://daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13" gracePeriod=45
Oct 30 19:17:50.097609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.097483    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": dial tcp 10.133.0.86:8000: connect: connection refused"
Oct 30 19:17:50.097609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.097483    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused"
Oct 30 19:17:50.172816 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:50.172775    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:17:50.248623 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.248595    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:17:50.285512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.285472    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:17:50.338088 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.338045    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32606f69-4aba-496f-be5f-8070f31440d2-kube-api-access-ntbxj\" (UniqueName: \"kubernetes.io/projected/32606f69-4aba-496f-be5f-8070f31440d2-kube-api-access-ntbxj\") pod \"32606f69-4aba-496f-be5f-8070f31440d2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") "
Oct 30 19:17:50.338313 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.338287    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-home\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-home\") pod \"32606f69-4aba-496f-be5f-8070f31440d2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") "
Oct 30 19:17:50.338614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.338588    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-home" pod "32606f69-4aba-496f-be5f-8070f31440d2" (UID: "32606f69-4aba-496f-be5f-8070f31440d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.338804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.338775    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-tmp-dir\") pod \"32606f69-4aba-496f-be5f-8070f31440d2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") "
Oct 30 19:17:50.338884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.338839    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-model-cache\") pod \"32606f69-4aba-496f-be5f-8070f31440d2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") "
Oct 30 19:17:50.338884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.338867    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-dshm\") pod \"32606f69-4aba-496f-be5f-8070f31440d2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") "
Oct 30 19:17:50.338999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.338937    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32606f69-4aba-496f-be5f-8070f31440d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/32606f69-4aba-496f-be5f-8070f31440d2-tls-certs\") pod \"32606f69-4aba-496f-be5f-8070f31440d2\" (UID: \"32606f69-4aba-496f-be5f-8070f31440d2\") "
Oct 30 19:17:50.339388 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.339293    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-model-cache" pod "32606f69-4aba-496f-be5f-8070f31440d2" (UID: "32606f69-4aba-496f-be5f-8070f31440d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.339388 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.339304    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-tmp-dir" pod "32606f69-4aba-496f-be5f-8070f31440d2" (UID: "32606f69-4aba-496f-be5f-8070f31440d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.340035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.340011    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.340177 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.340162    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.340287 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.340274    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.341542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.341477    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32606f69-4aba-496f-be5f-8070f31440d2-kube-api-access-ntbxj" pod "32606f69-4aba-496f-be5f-8070f31440d2" (UID: "32606f69-4aba-496f-be5f-8070f31440d2"). InnerVolumeSpecName "kube-api-access-ntbxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:50.342303 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.342278    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32606f69-4aba-496f-be5f-8070f31440d2-tls-certs" pod "32606f69-4aba-496f-be5f-8070f31440d2" (UID: "32606f69-4aba-496f-be5f-8070f31440d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:50.342870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.342835    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-dshm" pod "32606f69-4aba-496f-be5f-8070f31440d2" (UID: "32606f69-4aba-496f-be5f-8070f31440d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.441165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.441128    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-home\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-home\") pod \"b650c63a-b609-4a9c-83df-6dd942456b5d\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") "
Oct 30 19:17:50.441359 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.441185    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b650c63a-b609-4a9c-83df-6dd942456b5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b650c63a-b609-4a9c-83df-6dd942456b5d-tls-certs\") pod \"b650c63a-b609-4a9c-83df-6dd942456b5d\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") "
Oct 30 19:17:50.441359 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.441316    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b650c63a-b609-4a9c-83df-6dd942456b5d-kube-api-access-mk85b\" (UniqueName: \"kubernetes.io/projected/b650c63a-b609-4a9c-83df-6dd942456b5d-kube-api-access-mk85b\") pod \"b650c63a-b609-4a9c-83df-6dd942456b5d\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") "
Oct 30 19:17:50.441442 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.441414    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-tmp-dir\") pod \"b650c63a-b609-4a9c-83df-6dd942456b5d\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") "
Oct 30 19:17:50.441474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.441451    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-home" pod "b650c63a-b609-4a9c-83df-6dd942456b5d" (UID: "b650c63a-b609-4a9c-83df-6dd942456b5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.441474 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.441460    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-dshm\") pod \"b650c63a-b609-4a9c-83df-6dd942456b5d\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") "
Oct 30 19:17:50.441834 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.441800    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-tmp-dir" pod "b650c63a-b609-4a9c-83df-6dd942456b5d" (UID: "b650c63a-b609-4a9c-83df-6dd942456b5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.441986 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.441846    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-model-cache\") pod \"b650c63a-b609-4a9c-83df-6dd942456b5d\" (UID: \"b650c63a-b609-4a9c-83df-6dd942456b5d\") "
Oct 30 19:17:50.442121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.442096    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-model-cache" pod "b650c63a-b609-4a9c-83df-6dd942456b5d" (UID: "b650c63a-b609-4a9c-83df-6dd942456b5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.442543 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.442519    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.442638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.442549    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.442638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.442565    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32606f69-4aba-496f-be5f-8070f31440d2-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.442638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.442582    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32606f69-4aba-496f-be5f-8070f31440d2-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.442638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.442597    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntbxj\" (UniqueName: \"kubernetes.io/projected/32606f69-4aba-496f-be5f-8070f31440d2-kube-api-access-ntbxj\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.442638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.442613    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.443750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.443730    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-dshm" pod "b650c63a-b609-4a9c-83df-6dd942456b5d" (UID: "b650c63a-b609-4a9c-83df-6dd942456b5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:50.443852 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.443797    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b650c63a-b609-4a9c-83df-6dd942456b5d-kube-api-access-mk85b" pod "b650c63a-b609-4a9c-83df-6dd942456b5d" (UID: "b650c63a-b609-4a9c-83df-6dd942456b5d"). InnerVolumeSpecName "kube-api-access-mk85b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:50.443984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.443967    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b650c63a-b609-4a9c-83df-6dd942456b5d-tls-certs" pod "b650c63a-b609-4a9c-83df-6dd942456b5d" (UID: "b650c63a-b609-4a9c-83df-6dd942456b5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:50.543907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.543865    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b650c63a-b609-4a9c-83df-6dd942456b5d-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.543907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.543907    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk85b\" (UniqueName: \"kubernetes.io/projected/b650c63a-b609-4a9c-83df-6dd942456b5d-kube-api-access-mk85b\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.544167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.543922    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b650c63a-b609-4a9c-83df-6dd942456b5d-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:50.562831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.562802    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"
Oct 30 19:17:50.562831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.562820    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj" event={"ID":"81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa","Type":"ContainerDied","Data":"62d50155f3d6106b68a1fbef93c3308ec9996f6d2be304eaca8d49cbf1f856e4"}
Oct 30 19:17:50.563100 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.562895    4267 scope.go:122] "RemoveContainer" containerID="b06dee0f36c748139250a94141aa7a998af464415d329827dfa0b8fc2b6d87ee"
Oct 30 19:17:50.564648 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.564620    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb" event={"ID":"662447d4-f16d-458e-ba57-ce7cc87bcc8c","Type":"ContainerStarted","Data":"663c8f4474624769f7ea38f361efbdac1e195f2de7edc1e6cc550bb692cafeb2"}
Oct 30 19:17:50.564773 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.564671    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb" event={"ID":"662447d4-f16d-458e-ba57-ce7cc87bcc8c","Type":"ContainerStarted","Data":"4184809cc9c9d4453297b85840381ac9e255a4013cd19156947ebd0250080929"}
Oct 30 19:17:50.566417 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.566396    4267 generic.go:356] "Generic (PLEG): container finished" podID="32606f69-4aba-496f-be5f-8070f31440d2" containerID="021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f" exitCode=0
Oct 30 19:17:50.566582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.566448    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"
Oct 30 19:17:50.566582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.566482    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" event={"ID":"32606f69-4aba-496f-be5f-8070f31440d2","Type":"ContainerDied","Data":"021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f"}
Oct 30 19:17:50.566582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.566556    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2" event={"ID":"32606f69-4aba-496f-be5f-8070f31440d2","Type":"ContainerDied","Data":"d023d28764a5229e1c81e260f79d92322f36c536b5bd346f43a8ecc98ebeda71"}
Oct 30 19:17:50.567868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.567846    4267 generic.go:356] "Generic (PLEG): container finished" podID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerID="daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13" exitCode=0
Oct 30 19:17:50.568039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.567880    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"
Oct 30 19:17:50.568039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.567924    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" event={"ID":"b650c63a-b609-4a9c-83df-6dd942456b5d","Type":"ContainerDied","Data":"daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13"}
Oct 30 19:17:50.568039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.567956    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj" event={"ID":"b650c63a-b609-4a9c-83df-6dd942456b5d","Type":"ContainerDied","Data":"7229d07686ec29cde00336b8536f295848e008ed86d9140d3a4f3908bdb271d8"}
Oct 30 19:17:50.570025 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.569994    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx" event={"ID":"c1162287-d097-4762-a3b6-28e28c7910e1","Type":"ContainerStarted","Data":"c8d7e3d899baeb557a57028d9f05a730fef0516d1ad0a4f1fb2394bb0b63d007"}
Oct 30 19:17:50.575145 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.575121    4267 scope.go:122] "RemoveContainer" containerID="021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f"
Oct 30 19:17:50.586798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.586776    4267 scope.go:122] "RemoveContainer" containerID="021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f"
Oct 30 19:17:50.591016 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:50.590957    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f\": container with ID starting with 021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f not found: ID does not exist" containerID="021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f"
Oct 30 19:17:50.591135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.591031    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f"} err="failed to get container status \"021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f\": rpc error: code = NotFound desc = could not find container \"021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f\": container with ID starting with 021f0172058af51472130aae1c61161d7b5b7e0f7792b463667598fd6119069f not found: ID does not exist"
Oct 30 19:17:50.591135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.591103    4267 scope.go:122] "RemoveContainer" containerID="daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13"
Oct 30 19:17:50.601377 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.601356    4267 scope.go:122] "RemoveContainer" containerID="daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13"
Oct 30 19:17:50.601692 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:17:50.601671    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13\": container with ID starting with daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13 not found: ID does not exist" containerID="daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13"
Oct 30 19:17:50.601755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.601701    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13"} err="failed to get container status \"daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13\": rpc error: code = NotFound desc = could not find container \"daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13\": container with ID starting with daa8b294412218ce4feca9c0db04a322f170d935a0f50f752d61cd46235aff13 not found: ID does not exist"
Oct 30 19:17:50.730790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.730759    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"]
Oct 30 19:17:50.736001 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.735966    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/distinct-w1-975cf5-kserve-8f54c54b5-m6msj"]
Oct 30 19:17:50.843406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.843340    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb" podStartSLOduration=2.843323186 podStartE2EDuration="2.843323186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:50.840585328 +0000 UTC m=+2361.728502418" watchObservedRunningTime="2026-10-30 19:17:50.843323186 +0000 UTC m=+2361.731240275"
Oct 30 19:17:50.956012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:50.955940    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx" podStartSLOduration=2.955922512 podStartE2EDuration="2.955922512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:50.953115218 +0000 UTC m=+2361.841032308" watchObservedRunningTime="2026-10-30 19:17:50.955922512 +0000 UTC m=+2361.843839601"
Oct 30 19:17:51.065621 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.065588    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"]
Oct 30 19:17:51.068857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.068832    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-41db7c-659d876469-hk4pj"]
Oct 30 19:17:51.177547 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.177477    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"]
Oct 30 19:17:51.181582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.181548    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-975cf5/premium-w1-975cf5-kserve-76c95f7454-98ls2"]
Oct 30 19:17:51.221386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.221352    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32606f69-4aba-496f-be5f-8070f31440d2" path="/var/lib/kubelet/pods/32606f69-4aba-496f-be5f-8070f31440d2/volumes"
Oct 30 19:17:51.221842 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.221826    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa" path="/var/lib/kubelet/pods/81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa/volumes"
Oct 30 19:17:51.222215 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.222203    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" path="/var/lib/kubelet/pods/b650c63a-b609-4a9c-83df-6dd942456b5d/volumes"
Oct 30 19:17:51.580320 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.580287    4267 generic.go:356] "Generic (PLEG): container finished" podID="c1162287-d097-4762-a3b6-28e28c7910e1" containerID="c8d7e3d899baeb557a57028d9f05a730fef0516d1ad0a4f1fb2394bb0b63d007" exitCode=0
Oct 30 19:17:51.580549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.580360    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx" event={"ID":"c1162287-d097-4762-a3b6-28e28c7910e1","Type":"ContainerDied","Data":"c8d7e3d899baeb557a57028d9f05a730fef0516d1ad0a4f1fb2394bb0b63d007"}
Oct 30 19:17:51.583077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.583052    4267 generic.go:356] "Generic (PLEG): container finished" podID="662447d4-f16d-458e-ba57-ce7cc87bcc8c" containerID="663c8f4474624769f7ea38f361efbdac1e195f2de7edc1e6cc550bb692cafeb2" exitCode=0
Oct 30 19:17:51.583198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:51.583102    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb" event={"ID":"662447d4-f16d-458e-ba57-ce7cc87bcc8c","Type":"ContainerDied","Data":"663c8f4474624769f7ea38f361efbdac1e195f2de7edc1e6cc550bb692cafeb2"}
Oct 30 19:17:52.792949 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.792921    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:52.794886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.794869    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:52.819678 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.819649    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r" podUID="4547d45a-763d-485c-9c2f-9433013e97af" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:52.873746 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.873674    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662447d4-f16d-458e-ba57-ce7cc87bcc8c-kube-api-access-2jvjj\" (UniqueName: \"kubernetes.io/projected/662447d4-f16d-458e-ba57-ce7cc87bcc8c-kube-api-access-2jvjj\") pod \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") "
Oct 30 19:17:52.873862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.873802    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-xks-ca\") pod \"c1162287-d097-4762-a3b6-28e28c7910e1\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") "
Oct 30 19:17:52.873862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.873855    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-openshift-ca\") pod \"c1162287-d097-4762-a3b6-28e28c7910e1\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") "
Oct 30 19:17:52.873943 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.873883    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-openshift-ca\") pod \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") "
Oct 30 19:17:52.874033 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.874012    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1162287-d097-4762-a3b6-28e28c7910e1-kube-api-access-4grzb\" (UniqueName: \"kubernetes.io/projected/c1162287-d097-4762-a3b6-28e28c7910e1-kube-api-access-4grzb\") pod \"c1162287-d097-4762-a3b6-28e28c7910e1\" (UID: \"c1162287-d097-4762-a3b6-28e28c7910e1\") "
Oct 30 19:17:52.874106 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.874061    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-xks-ca\") pod \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\" (UID: \"662447d4-f16d-458e-ba57-ce7cc87bcc8c\") "
Oct 30 19:17:52.874230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.874209    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-openshift-ca" pod "662447d4-f16d-458e-ba57-ce7cc87bcc8c" (UID: "662447d4-f16d-458e-ba57-ce7cc87bcc8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:52.874280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.874234    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-openshift-ca" pod "c1162287-d097-4762-a3b6-28e28c7910e1" (UID: "c1162287-d097-4762-a3b6-28e28c7910e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:52.874586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.874557    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.874586 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.874581    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.876144 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.876121    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662447d4-f16d-458e-ba57-ce7cc87bcc8c-kube-api-access-2jvjj" pod "662447d4-f16d-458e-ba57-ce7cc87bcc8c" (UID: "662447d4-f16d-458e-ba57-ce7cc87bcc8c"). InnerVolumeSpecName "kube-api-access-2jvjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:52.876230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.876174    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1162287-d097-4762-a3b6-28e28c7910e1-kube-api-access-4grzb" pod "c1162287-d097-4762-a3b6-28e28c7910e1" (UID: "c1162287-d097-4762-a3b6-28e28c7910e1"). InnerVolumeSpecName "kube-api-access-4grzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:52.876420 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.876395    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-xks-ca" pod "662447d4-f16d-458e-ba57-ce7cc87bcc8c" (UID: "662447d4-f16d-458e-ba57-ce7cc87bcc8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:52.876420 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.876411    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-xks-ca" pod "c1162287-d097-4762-a3b6-28e28c7910e1" (UID: "c1162287-d097-4762-a3b6-28e28c7910e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:52.976372 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.976335    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1162287-d097-4762-a3b6-28e28c7910e1-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.976570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.976377    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4grzb\" (UniqueName: \"kubernetes.io/projected/c1162287-d097-4762-a3b6-28e28c7910e1-kube-api-access-4grzb\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.976570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.976394    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/662447d4-f16d-458e-ba57-ce7cc87bcc8c-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.976570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:52.976409    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvjj\" (UniqueName: \"kubernetes.io/projected/662447d4-f16d-458e-ba57-ce7cc87bcc8c-kube-api-access-2jvjj\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:53.598618 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:53.598535    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"
Oct 30 19:17:53.598618 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:53.598582    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx" event={"ID":"c1162287-d097-4762-a3b6-28e28c7910e1","Type":"ContainerDied","Data":"432f7e70563c3da0848f7945b8a24c9480ff828694ac7643010b67b3635ebc05"}
Oct 30 19:17:53.598828 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:53.598639    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="432f7e70563c3da0848f7945b8a24c9480ff828694ac7643010b67b3635ebc05"
Oct 30 19:17:53.600293 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:53.600264    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"
Oct 30 19:17:53.600408 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:53.600314    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb" event={"ID":"662447d4-f16d-458e-ba57-ce7cc87bcc8c","Type":"ContainerDied","Data":"4184809cc9c9d4453297b85840381ac9e255a4013cd19156947ebd0250080929"}
Oct 30 19:17:53.600408 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:17:53.600361    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4184809cc9c9d4453297b85840381ac9e255a4013cd19156947ebd0250080929"
Oct 30 19:18:01.685745 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:01.685698    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" containerID="cri-o://38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136" gracePeriod=45
Oct 30 19:18:01.977405 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:01.977375    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:18:02.062048 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062012    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tmp-dir\") pod \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") "
Oct 30 19:18:02.062269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062071    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tls-certs\") pod \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") "
Oct 30 19:18:02.062269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062094    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-model-cache\") pod \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") "
Oct 30 19:18:02.062269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062213    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-home\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-home\") pod \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") "
Oct 30 19:18:02.062462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062272    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-dshm\") pod \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") "
Oct 30 19:18:02.062462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062303    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-kube-api-access-rm6s7\" (UniqueName: \"kubernetes.io/projected/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-kube-api-access-rm6s7\") pod \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\" (UID: \"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc\") "
Oct 30 19:18:02.062462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062363    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tmp-dir" pod "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" (UID: "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:02.062462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062413    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-model-cache" pod "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" (UID: "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:02.062682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062561    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-home" pod "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" (UID: "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:02.062942 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062919    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:02.063038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062944    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:02.063038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.062958    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:02.064821 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.064796    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-dshm" pod "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" (UID: "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:02.064924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.064884    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tls-certs" pod "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" (UID: "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:02.064924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.064884    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-kube-api-access-rm6s7" pod "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" (UID: "e44de0b9-15e4-4b2c-91ba-3f1d57f198bc"). InnerVolumeSpecName "kube-api-access-rm6s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:02.164358 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.164317    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:02.164358 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.164358    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm6s7\" (UniqueName: \"kubernetes.io/projected/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-kube-api-access-rm6s7\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:02.164643 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.164376    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:02.723213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.723169    4267 generic.go:356] "Generic (PLEG): container finished" podID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerID="38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136" exitCode=0
Oct 30 19:18:02.723722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.723234    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"
Oct 30 19:18:02.723722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.723259    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" event={"ID":"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc","Type":"ContainerDied","Data":"38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136"}
Oct 30 19:18:02.723722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.723326    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" event={"ID":"e44de0b9-15e4-4b2c-91ba-3f1d57f198bc","Type":"ContainerDied","Data":"9c5fa0f062510c46e77d2a009605864370d1a94ab604b21c848bec12ebb377f8"}
Oct 30 19:18:02.723722 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.723349    4267 scope.go:122] "RemoveContainer" containerID="38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136"
Oct 30 19:18:02.736075 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.736054    4267 scope.go:122] "RemoveContainer" containerID="38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136"
Oct 30 19:18:02.736409 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:18:02.736388    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136\": container with ID starting with 38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136 not found: ID does not exist" containerID="38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136"
Oct 30 19:18:02.736469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.736420    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136"} err="failed to get container status \"38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136\": rpc error: code = NotFound desc = could not find container \"38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136\": container with ID starting with 38fa6b029415cd9bc5bb505582d6d4cececd814bb5b54771b572f9a591642136 not found: ID does not exist"
Oct 30 19:18:02.819940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.819902    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r" podUID="4547d45a-763d-485c-9c2f-9433013e97af" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:18:02.891781 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.891750    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"]
Oct 30 19:18:02.932360 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.932319    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml"]
Oct 30 19:18:02.972159 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:02.972112    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1290a6/distinct-w4-1290a6-kserve-75d56d9d7b-jpwml" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:18:03.221023 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:03.220990    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" path="/var/lib/kubelet/pods/e44de0b9-15e4-4b2c-91ba-3f1d57f198bc/volumes"
Oct 30 19:18:06.346710 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.346678    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 19:18:06.423030 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.423001    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4547d45a-763d-485c-9c2f-9433013e97af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4547d45a-763d-485c-9c2f-9433013e97af-plugins-config-volume\") pod \"4547d45a-763d-485c-9c2f-9433013e97af\" (UID: \"4547d45a-763d-485c-9c2f-9433013e97af\") "
Oct 30 19:18:06.423211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.423074    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4547d45a-763d-485c-9c2f-9433013e97af-kube-api-access-2p47c\" (UniqueName: \"kubernetes.io/projected/4547d45a-763d-485c-9c2f-9433013e97af-kube-api-access-2p47c\") pod \"4547d45a-763d-485c-9c2f-9433013e97af\" (UID: \"4547d45a-763d-485c-9c2f-9433013e97af\") "
Oct 30 19:18:06.423476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.423451    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4547d45a-763d-485c-9c2f-9433013e97af-plugins-config-volume" pod "4547d45a-763d-485c-9c2f-9433013e97af" (UID: "4547d45a-763d-485c-9c2f-9433013e97af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:06.425168 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.425151    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4547d45a-763d-485c-9c2f-9433013e97af-kube-api-access-2p47c" pod "4547d45a-763d-485c-9c2f-9433013e97af" (UID: "4547d45a-763d-485c-9c2f-9433013e97af"). InnerVolumeSpecName "kube-api-access-2p47c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:06.524405 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.524370    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4547d45a-763d-485c-9c2f-9433013e97af-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:06.524405 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.524401    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p47c\" (UniqueName: \"kubernetes.io/projected/4547d45a-763d-485c-9c2f-9433013e97af-kube-api-access-2p47c\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:06.754770 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.754730    4267 generic.go:356] "Generic (PLEG): container finished" podID="4547d45a-763d-485c-9c2f-9433013e97af" containerID="d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160" exitCode=0
Oct 30 19:18:06.754934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.754783    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"
Oct 30 19:18:06.754934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.754809    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r" event={"ID":"4547d45a-763d-485c-9c2f-9433013e97af","Type":"ContainerDied","Data":"d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160"}
Oct 30 19:18:06.754934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.754867    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r" event={"ID":"4547d45a-763d-485c-9c2f-9433013e97af","Type":"ContainerDied","Data":"43cf957842961aec9465083755729cc707b42ee11fd5a3ffacdbfee75db6aefe"}
Oct 30 19:18:06.754934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.754891    4267 scope.go:122] "RemoveContainer" containerID="d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160"
Oct 30 19:18:06.765190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.765172    4267 scope.go:122] "RemoveContainer" containerID="d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160"
Oct 30 19:18:06.765463 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:18:06.765444    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160\": container with ID starting with d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160 not found: ID does not exist" containerID="d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160"
Oct 30 19:18:06.765579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.765475    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160"} err="failed to get container status \"d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160\": rpc error: code = NotFound desc = could not find container \"d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160\": container with ID starting with d6f98661db2bc165899106ac424e6015dd93fb6af5e0a51a6d0c3f8333a51160 not found: ID does not exist"
Oct 30 19:18:06.919520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.919432    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"]
Oct 30 19:18:06.969295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:06.969261    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-975cf5-f474cb955-h8t2r"]
Oct 30 19:18:07.223165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:07.223126    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4547d45a-763d-485c-9c2f-9433013e97af" path="/var/lib/kubelet/pods/4547d45a-763d-485c-9c2f-9433013e97af/volumes"
Oct 30 19:18:15.171074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:15.170986    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"]
Oct 30 19:18:15.176020 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:15.175989    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-903a82-gmscw"]
Oct 30 19:18:15.221336 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:15.221299    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92408e1f-b0f2-4d10-9899-71656d977ed6" path="/var/lib/kubelet/pods/92408e1f-b0f2-4d10-9899-71656d977ed6/volumes"
Oct 30 19:18:19.385065 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385028    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"]
Oct 30 19:18:19.385528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385513    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c1162287-d097-4762-a3b6-28e28c7910e1" containerName="revoke-keys"
Oct 30 19:18:19.385582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385528    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1162287-d097-4762-a3b6-28e28c7910e1" containerName="revoke-keys"
Oct 30 19:18:19.385582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385544    4267 cpu_manager.go:414] "containerMap: removing container" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main"
Oct 30 19:18:19.385582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385549    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main"
Oct 30 19:18:19.385582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385568    4267 cpu_manager.go:414] "containerMap: removing container" podUID="4547d45a-763d-485c-9c2f-9433013e97af" containerName="payload-pre-processing"
Oct 30 19:18:19.385582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385576    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4547d45a-763d-485c-9c2f-9433013e97af" containerName="payload-pre-processing"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385596    4267 cpu_manager.go:414] "containerMap: removing container" podUID="662447d4-f16d-458e-ba57-ce7cc87bcc8c" containerName="revoke-keys"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385601    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662447d4-f16d-458e-ba57-ce7cc87bcc8c" containerName="revoke-keys"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385618    4267 cpu_manager.go:414] "containerMap: removing container" podUID="81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa" containerName="payload-processing"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385623    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa" containerName="payload-processing"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385637    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385644    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385655    4267 cpu_manager.go:414] "containerMap: removing container" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385660    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385737    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b650c63a-b609-4a9c-83df-6dd942456b5d" containerName="main"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385746    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44de0b9-15e4-4b2c-91ba-3f1d57f198bc" containerName="main"
Oct 30 19:18:19.385755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385754    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1162287-d097-4762-a3b6-28e28c7910e1" containerName="revoke-keys"
Oct 30 19:18:19.386135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385762    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="32606f69-4aba-496f-be5f-8070f31440d2" containerName="main"
Oct 30 19:18:19.386135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385769    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="4547d45a-763d-485c-9c2f-9433013e97af" containerName="payload-pre-processing"
Oct 30 19:18:19.386135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385777    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="662447d4-f16d-458e-ba57-ce7cc87bcc8c" containerName="revoke-keys"
Oct 30 19:18:19.386135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.385785    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a5e580-a4ab-4fd7-8b9a-a92367ddeaaa" containerName="payload-processing"
Oct 30 19:18:19.390422 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.390402    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:19.446438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.446406    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"]
Oct 30 19:18:19.491858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.491798    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:18:19.491858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.491836    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:18:19.536511 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.536458    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88a2f0cc-6d6g9\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:19.536713 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.536572    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24jrp\" (UniqueName: \"kubernetes.io/projected/e5d641d9-e712-4a48-ad72-b0c2443b309d-kube-api-access-24jrp\") pod \"maas-api-revoke-sub-88a2f0cc-6d6g9\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:19.536713 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.536669    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88a2f0cc-6d6g9\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:19.638339 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.638253    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88a2f0cc-6d6g9\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:19.638339 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.638327    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24jrp\" (UniqueName: \"kubernetes.io/projected/e5d641d9-e712-4a48-ad72-b0c2443b309d-kube-api-access-24jrp\") pod \"maas-api-revoke-sub-88a2f0cc-6d6g9\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:19.638576 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.638446    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88a2f0cc-6d6g9\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:19.638988 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.638954    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88a2f0cc-6d6g9\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:19.641043 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.641022    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88a2f0cc-6d6g9\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:19.695164 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.695128    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24jrp\" (UniqueName: \"kubernetes.io/projected/e5d641d9-e712-4a48-ad72-b0c2443b309d-kube-api-access-24jrp\") pod \"maas-api-revoke-sub-88a2f0cc-6d6g9\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:19.703883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:19.703848    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:20.081253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:20.081221    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"]
Oct 30 19:18:20.081929 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:18:20.081904    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d641d9_e712_4a48_ad72_b0c2443b309d.slice/crio-34ac50a88122c5c8167c71a42252a923e6fdbc1a2e8e2e6a5e4319ffee5fca0e WatchSource:0}: Error finding container 34ac50a88122c5c8167c71a42252a923e6fdbc1a2e8e2e6a5e4319ffee5fca0e: Status 404 returned error can't find the container with id 34ac50a88122c5c8167c71a42252a923e6fdbc1a2e8e2e6a5e4319ffee5fca0e
Oct 30 19:18:20.159581 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:18:20.159539    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:18:20.859753 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:20.859715    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9" event={"ID":"e5d641d9-e712-4a48-ad72-b0c2443b309d","Type":"ContainerStarted","Data":"76ac98888cec4d9f91863b43276dc04f9ef029cc64e817577abe1dcd5fd79047"}
Oct 30 19:18:20.860158 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:20.859767    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9" event={"ID":"e5d641d9-e712-4a48-ad72-b0c2443b309d","Type":"ContainerStarted","Data":"34ac50a88122c5c8167c71a42252a923e6fdbc1a2e8e2e6a5e4319ffee5fca0e"}
Oct 30 19:18:20.974968 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:20.974915    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9" podStartSLOduration=1.9748965539999999 podStartE2EDuration="1.974896554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:20.973034099 +0000 UTC m=+2391.860951190" watchObservedRunningTime="2026-10-30 19:18:20.974896554 +0000 UTC m=+2391.862813644"
Oct 30 19:18:21.868170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:21.868133    4267 generic.go:356] "Generic (PLEG): container finished" podID="e5d641d9-e712-4a48-ad72-b0c2443b309d" containerID="76ac98888cec4d9f91863b43276dc04f9ef029cc64e817577abe1dcd5fd79047" exitCode=0
Oct 30 19:18:21.868630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:21.868222    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9" event={"ID":"e5d641d9-e712-4a48-ad72-b0c2443b309d","Type":"ContainerDied","Data":"76ac98888cec4d9f91863b43276dc04f9ef029cc64e817577abe1dcd5fd79047"}
Oct 30 19:18:23.121604 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.121579    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:23.192804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.192771    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d641d9-e712-4a48-ad72-b0c2443b309d-kube-api-access-24jrp\" (UniqueName: \"kubernetes.io/projected/e5d641d9-e712-4a48-ad72-b0c2443b309d-kube-api-access-24jrp\") pod \"e5d641d9-e712-4a48-ad72-b0c2443b309d\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") "
Oct 30 19:18:23.193110 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.192841    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-xks-ca\") pod \"e5d641d9-e712-4a48-ad72-b0c2443b309d\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") "
Oct 30 19:18:23.193110 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.192899    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-openshift-ca\") pod \"e5d641d9-e712-4a48-ad72-b0c2443b309d\" (UID: \"e5d641d9-e712-4a48-ad72-b0c2443b309d\") "
Oct 30 19:18:23.193288 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.193263    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-openshift-ca" pod "e5d641d9-e712-4a48-ad72-b0c2443b309d" (UID: "e5d641d9-e712-4a48-ad72-b0c2443b309d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:23.193461 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.193436    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:23.195321 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.195293    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-xks-ca" pod "e5d641d9-e712-4a48-ad72-b0c2443b309d" (UID: "e5d641d9-e712-4a48-ad72-b0c2443b309d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:23.195523 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.195481    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d641d9-e712-4a48-ad72-b0c2443b309d-kube-api-access-24jrp" pod "e5d641d9-e712-4a48-ad72-b0c2443b309d" (UID: "e5d641d9-e712-4a48-ad72-b0c2443b309d"). InnerVolumeSpecName "kube-api-access-24jrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:23.294825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.294630    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d641d9-e712-4a48-ad72-b0c2443b309d-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:23.294825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.294674    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24jrp\" (UniqueName: \"kubernetes.io/projected/e5d641d9-e712-4a48-ad72-b0c2443b309d-kube-api-access-24jrp\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:23.892145 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.892113    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"
Oct 30 19:18:23.892145 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.892137    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9" event={"ID":"e5d641d9-e712-4a48-ad72-b0c2443b309d","Type":"ContainerDied","Data":"34ac50a88122c5c8167c71a42252a923e6fdbc1a2e8e2e6a5e4319ffee5fca0e"}
Oct 30 19:18:23.892407 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:23.892192    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34ac50a88122c5c8167c71a42252a923e6fdbc1a2e8e2e6a5e4319ffee5fca0e"
Oct 30 19:18:30.050138 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.050086    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"]
Oct 30 19:18:30.050669 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.050651    4267 cpu_manager.go:414] "containerMap: removing container" podUID="e5d641d9-e712-4a48-ad72-b0c2443b309d" containerName="revoke-keys"
Oct 30 19:18:30.050669 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.050670    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d641d9-e712-4a48-ad72-b0c2443b309d" containerName="revoke-keys"
Oct 30 19:18:30.050803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.050779    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d641d9-e712-4a48-ad72-b0c2443b309d" containerName="revoke-keys"
Oct 30 19:18:30.054126 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.054103    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.081714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.081673    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqkc\" (UniqueName: \"kubernetes.io/projected/d304bea0-f8b8-43b5-a243-c49592c152ac-kube-api-access-5kqkc\") pod \"maas-api-revoke-sub-cc8d3955-wr2fq\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.081887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.081784    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc8d3955-wr2fq\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.081887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.081812    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc8d3955-wr2fq\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.111967 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.111930    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"]
Oct 30 19:18:30.156588 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.156554    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:18:30.156784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.156748    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:18:30.182639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.182597    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc8d3955-wr2fq\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.182639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.182653    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc8d3955-wr2fq\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.182947 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.182907    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqkc\" (UniqueName: \"kubernetes.io/projected/d304bea0-f8b8-43b5-a243-c49592c152ac-kube-api-access-5kqkc\") pod \"maas-api-revoke-sub-cc8d3955-wr2fq\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.183427 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.183401    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc8d3955-wr2fq\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.185453 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.185372    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc8d3955-wr2fq\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.239921 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.239882    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqkc\" (UniqueName: \"kubernetes.io/projected/d304bea0-f8b8-43b5-a243-c49592c152ac-kube-api-access-5kqkc\") pod \"maas-api-revoke-sub-cc8d3955-wr2fq\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.370233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.370139    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:30.751462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.751423    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"]
Oct 30 19:18:30.752853 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:18:30.752824    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd304bea0_f8b8_43b5_a243_c49592c152ac.slice/crio-97a5661cef49066f82e661ffed5826ffc0459d7a94a7c415e70d7ad7f8cc5429 WatchSource:0}: Error finding container 97a5661cef49066f82e661ffed5826ffc0459d7a94a7c415e70d7ad7f8cc5429: Status 404 returned error can't find the container with id 97a5661cef49066f82e661ffed5826ffc0459d7a94a7c415e70d7ad7f8cc5429
Oct 30 19:18:30.943625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:30.943577    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq" event={"ID":"d304bea0-f8b8-43b5-a243-c49592c152ac","Type":"ContainerStarted","Data":"97a5661cef49066f82e661ffed5826ffc0459d7a94a7c415e70d7ad7f8cc5429"}
Oct 30 19:18:31.353216 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:31.353137    4267 scope.go:122] "RemoveContainer" containerID="cfb669fe45a432447e456da34ba768b2a40bd795f5904b75a03f4892a953718b"
Oct 30 19:18:31.367485 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:31.367462    4267 scope.go:122] "RemoveContainer" containerID="84e9853b268c8acc6b14234a982b51d0aca9574a9646569d39ed90c4ee248164"
Oct 30 19:18:31.954067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:31.954031    4267 generic.go:356] "Generic (PLEG): container finished" podID="d304bea0-f8b8-43b5-a243-c49592c152ac" containerID="c2bd0abde857e5e6d724d2159f35a4d1bc6da64116b923194d2d47929356239a" exitCode=0
Oct 30 19:18:31.954270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:31.954102    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq" event={"ID":"d304bea0-f8b8-43b5-a243-c49592c152ac","Type":"ContainerDied","Data":"c2bd0abde857e5e6d724d2159f35a4d1bc6da64116b923194d2d47929356239a"}
Oct 30 19:18:33.105321 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.105295    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:33.124186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.123787    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d304bea0-f8b8-43b5-a243-c49592c152ac-kube-api-access-5kqkc\" (UniqueName: \"kubernetes.io/projected/d304bea0-f8b8-43b5-a243-c49592c152ac-kube-api-access-5kqkc\") pod \"d304bea0-f8b8-43b5-a243-c49592c152ac\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") "
Oct 30 19:18:33.124186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.123967    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-xks-ca\") pod \"d304bea0-f8b8-43b5-a243-c49592c152ac\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") "
Oct 30 19:18:33.124186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.124100    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-openshift-ca\") pod \"d304bea0-f8b8-43b5-a243-c49592c152ac\" (UID: \"d304bea0-f8b8-43b5-a243-c49592c152ac\") "
Oct 30 19:18:33.124971 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.124929    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-openshift-ca" pod "d304bea0-f8b8-43b5-a243-c49592c152ac" (UID: "d304bea0-f8b8-43b5-a243-c49592c152ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:33.126240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.126205    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d304bea0-f8b8-43b5-a243-c49592c152ac-kube-api-access-5kqkc" pod "d304bea0-f8b8-43b5-a243-c49592c152ac" (UID: "d304bea0-f8b8-43b5-a243-c49592c152ac"). InnerVolumeSpecName "kube-api-access-5kqkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:33.126455 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.126439    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-xks-ca" pod "d304bea0-f8b8-43b5-a243-c49592c152ac" (UID: "d304bea0-f8b8-43b5-a243-c49592c152ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:33.225538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.225431    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:33.225538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.225472    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d304bea0-f8b8-43b5-a243-c49592c152ac-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:33.225538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.225487    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kqkc\" (UniqueName: \"kubernetes.io/projected/d304bea0-f8b8-43b5-a243-c49592c152ac-kube-api-access-5kqkc\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:33.968368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.968330    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"
Oct 30 19:18:33.968368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.968360    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq" event={"ID":"d304bea0-f8b8-43b5-a243-c49592c152ac","Type":"ContainerDied","Data":"97a5661cef49066f82e661ffed5826ffc0459d7a94a7c415e70d7ad7f8cc5429"}
Oct 30 19:18:33.968622 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:33.968407    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97a5661cef49066f82e661ffed5826ffc0459d7a94a7c415e70d7ad7f8cc5429"
Oct 30 19:18:34.119146 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:34.119108    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"]
Oct 30 19:18:35.985205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:35.985173    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"]
Oct 30 19:18:35.985701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:35.985477    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d" podUID="91d78de9-1580-4e7b-a3e3-e0a79f32dcf2" containerName="maas-api" containerID="cri-o://be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889" gracePeriod=30
Oct 30 19:18:36.301063 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.301032    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:18:36.415878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.415847    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-openshift-service-ca\") pod \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") "
Oct 30 19:18:36.416084 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.416055    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-kube-api-access-khwws\" (UniqueName: \"kubernetes.io/projected/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-kube-api-access-khwws\") pod \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") "
Oct 30 19:18:36.416132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.416082    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-metrics-certs\") pod \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") "
Oct 30 19:18:36.416132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.416106    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-maas-api-tls\") pod \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\" (UID: \"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2\") "
Oct 30 19:18:36.416226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.416197    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-openshift-service-ca" pod "91d78de9-1580-4e7b-a3e3-e0a79f32dcf2" (UID: "91d78de9-1580-4e7b-a3e3-e0a79f32dcf2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:36.416551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.416535    4267 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-openshift-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:36.418284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.418261    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-maas-api-tls" pod "91d78de9-1580-4e7b-a3e3-e0a79f32dcf2" (UID: "91d78de9-1580-4e7b-a3e3-e0a79f32dcf2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:36.418356 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.418276    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-metrics-certs" pod "91d78de9-1580-4e7b-a3e3-e0a79f32dcf2" (UID: "91d78de9-1580-4e7b-a3e3-e0a79f32dcf2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:36.418393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.418362    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-kube-api-access-khwws" pod "91d78de9-1580-4e7b-a3e3-e0a79f32dcf2" (UID: "91d78de9-1580-4e7b-a3e3-e0a79f32dcf2"). InnerVolumeSpecName "kube-api-access-khwws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:36.517366 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.517327    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khwws\" (UniqueName: \"kubernetes.io/projected/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-kube-api-access-khwws\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:36.517366 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.517362    4267 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-metrics-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:36.517366 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.517372    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2-maas-api-tls\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:36.960534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.956393    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"]
Oct 30 19:18:36.992437 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.992400    4267 generic.go:356] "Generic (PLEG): container finished" podID="91d78de9-1580-4e7b-a3e3-e0a79f32dcf2" containerID="be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889" exitCode=0
Oct 30 19:18:36.992915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.992441    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"
Oct 30 19:18:36.992915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.992480    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d" event={"ID":"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2","Type":"ContainerDied","Data":"be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889"}
Oct 30 19:18:36.992915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.992570    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d" event={"ID":"91d78de9-1580-4e7b-a3e3-e0a79f32dcf2","Type":"ContainerDied","Data":"41d357f24a881ce975ec118f3bf998e06f085e0077ae210d0c90bafab483925e"}
Oct 30 19:18:36.992915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:36.992593    4267 scope.go:122] "RemoveContainer" containerID="be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889"
Oct 30 19:18:37.001367 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:37.001179    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-41db7c-q8dhl"]
Oct 30 19:18:37.004427 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:37.004410    4267 scope.go:122] "RemoveContainer" containerID="be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889"
Oct 30 19:18:37.004783 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:18:37.004763    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889\": container with ID starting with be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889 not found: ID does not exist" containerID="be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889"
Oct 30 19:18:37.004850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:37.004794    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889"} err="failed to get container status \"be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889\": rpc error: code = NotFound desc = could not find container \"be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889\": container with ID starting with be775fd8770788dd2ea38b35180244184e2125cd88245389e60632ac74e01889 not found: ID does not exist"
Oct 30 19:18:37.157413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:37.157378    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"]
Oct 30 19:18:37.160470 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:37.160444    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36d676-6cbcf695fb-jjm6d"]
Oct 30 19:18:37.220599 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:37.220484    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c66545-4766-4956-ae0f-b416735b36ab" path="/var/lib/kubelet/pods/06c66545-4766-4956-ae0f-b416735b36ab/volumes"
Oct 30 19:18:37.221069 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:37.221047    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d78de9-1580-4e7b-a3e3-e0a79f32dcf2" path="/var/lib/kubelet/pods/91d78de9-1580-4e7b-a3e3-e0a79f32dcf2/volumes"
Oct 30 19:18:39.119966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:39.119901    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj" podUID="62062c97-8d55-4933-b39e-8961ed4925b9" containerName="payload-processing" containerID="cri-o://edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641" gracePeriod=25
Oct 30 19:18:39.373815 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:39.373757    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:18:39.457065 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:39.457026    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62062c97-8d55-4933-b39e-8961ed4925b9-kube-api-access-wnqb9\" (UniqueName: \"kubernetes.io/projected/62062c97-8d55-4933-b39e-8961ed4925b9-kube-api-access-wnqb9\") pod \"62062c97-8d55-4933-b39e-8961ed4925b9\" (UID: \"62062c97-8d55-4933-b39e-8961ed4925b9\") "
Oct 30 19:18:39.457264 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:39.457156    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62062c97-8d55-4933-b39e-8961ed4925b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62062c97-8d55-4933-b39e-8961ed4925b9-plugins-config-volume\") pod \"62062c97-8d55-4933-b39e-8961ed4925b9\" (UID: \"62062c97-8d55-4933-b39e-8961ed4925b9\") "
Oct 30 19:18:39.457553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:39.457525    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62062c97-8d55-4933-b39e-8961ed4925b9-plugins-config-volume" pod "62062c97-8d55-4933-b39e-8961ed4925b9" (UID: "62062c97-8d55-4933-b39e-8961ed4925b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:39.459138 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:39.459118    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62062c97-8d55-4933-b39e-8961ed4925b9-kube-api-access-wnqb9" pod "62062c97-8d55-4933-b39e-8961ed4925b9" (UID: "62062c97-8d55-4933-b39e-8961ed4925b9"). InnerVolumeSpecName "kube-api-access-wnqb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:39.558267 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:39.558229    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnqb9\" (UniqueName: \"kubernetes.io/projected/62062c97-8d55-4933-b39e-8961ed4925b9-kube-api-access-wnqb9\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:39.558267 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:39.558260    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62062c97-8d55-4933-b39e-8961ed4925b9-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:40.031172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:40.031137    4267 generic.go:356] "Generic (PLEG): container finished" podID="62062c97-8d55-4933-b39e-8961ed4925b9" containerID="edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641" exitCode=0
Oct 30 19:18:40.031378 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:40.031208    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"
Oct 30 19:18:40.031378 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:40.031215    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj" event={"ID":"62062c97-8d55-4933-b39e-8961ed4925b9","Type":"ContainerDied","Data":"edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641"}
Oct 30 19:18:40.031378 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:40.031298    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj" event={"ID":"62062c97-8d55-4933-b39e-8961ed4925b9","Type":"ContainerDied","Data":"ddde7acecb9735021410ae5d91360f533c33b2537fe903a077a66fe12f681e32"}
Oct 30 19:18:40.031378 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:40.031322    4267 scope.go:122] "RemoveContainer" containerID="edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641"
Oct 30 19:18:40.041638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:40.041613    4267 scope.go:122] "RemoveContainer" containerID="edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641"
Oct 30 19:18:40.041892 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:18:40.041874    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641\": container with ID starting with edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641 not found: ID does not exist" containerID="edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641"
Oct 30 19:18:40.041930 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:40.041902    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641"} err="failed to get container status \"edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641\": rpc error: code = NotFound desc = could not find container \"edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641\": container with ID starting with edd16d4b6ea73c60d899ccdb759a64fb55cd2445ce311e910a9034efba01c641 not found: ID does not exist"
Oct 30 19:18:40.198622 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:40.198585    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"]
Oct 30 19:18:40.248024 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:40.247990    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-36d676-858847d555-2bwrj"]
Oct 30 19:18:41.220641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:41.220610    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62062c97-8d55-4933-b39e-8961ed4925b9" path="/var/lib/kubelet/pods/62062c97-8d55-4933-b39e-8961ed4925b9/volumes"
Oct 30 19:18:46.012799 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:46.012765    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"]
Oct 30 19:18:46.013253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:46.012990    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" podUID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" containerName="istio-proxy" containerID="cri-o://2cbd31ddda16c23e84ddbf989935cc645f6caf1e6df6e878b0764bff8979398d" gracePeriod=30
Oct 30 19:18:46.033833 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:46.033793    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"]
Oct 30 19:18:46.034083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:46.034039    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerName="istio-proxy" containerID="cri-o://882d8eb23e86bf8429b530cfb89a32bcf8d9f111ca18e3c4d6c96000629fac5c" gracePeriod=30
Oct 30 19:18:46.057014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:46.056978    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"]
Oct 30 19:18:46.057439 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:46.057407    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerName="istio-proxy" containerID="cri-o://86626663480cd2f650d2043e2f8f86dc167e523ba5eb866233cf3e74a9383e4e" gracePeriod=30
Oct 30 19:18:50.166407 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:18:50.166365    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:18:51.179642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.179592    4267 generic.go:356] "Generic (PLEG): container finished" podID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerID="86626663480cd2f650d2043e2f8f86dc167e523ba5eb866233cf3e74a9383e4e" exitCode=0
Oct 30 19:18:51.180057 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.179666    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" event={"ID":"f04ad0d0-e089-4dce-8735-968a6eae12e5","Type":"ContainerDied","Data":"86626663480cd2f650d2043e2f8f86dc167e523ba5eb866233cf3e74a9383e4e"}
Oct 30 19:18:51.182372 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.182336    4267 generic.go:356] "Generic (PLEG): container finished" podID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerID="882d8eb23e86bf8429b530cfb89a32bcf8d9f111ca18e3c4d6c96000629fac5c" exitCode=0
Oct 30 19:18:51.182534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.182406    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" event={"ID":"34f3cf07-299e-485e-9bb7-ba2641d40834","Type":"ContainerDied","Data":"882d8eb23e86bf8429b530cfb89a32bcf8d9f111ca18e3c4d6c96000629fac5c"}
Oct 30 19:18:51.184625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.184595    4267 generic.go:356] "Generic (PLEG): container finished" podID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" containerID="2cbd31ddda16c23e84ddbf989935cc645f6caf1e6df6e878b0764bff8979398d" exitCode=0
Oct 30 19:18:51.184728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.184671    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" event={"ID":"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d","Type":"ContainerDied","Data":"2cbd31ddda16c23e84ddbf989935cc645f6caf1e6df6e878b0764bff8979398d"}
Oct 30 19:18:51.340271 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.340243    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:18:51.384166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.384141    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:18:51.390123 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.390101    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:18:51.408162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408136    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-socket\") pod \"34f3cf07-299e-485e-9bb7-ba2641d40834\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") "
Oct 30 19:18:51.408292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408278    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-credential-socket\") pod \"34f3cf07-299e-485e-9bb7-ba2641d40834\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") "
Oct 30 19:18:51.408352 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408335    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-podinfo\") pod \"34f3cf07-299e-485e-9bb7-ba2641d40834\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") "
Oct 30 19:18:51.408412 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408381    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-envoy\") pod \"34f3cf07-299e-485e-9bb7-ba2641d40834\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") "
Oct 30 19:18:51.408469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408415    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-socket" pod "34f3cf07-299e-485e-9bb7-ba2641d40834" (UID: "34f3cf07-299e-485e-9bb7-ba2641d40834"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.408469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408428    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f3cf07-299e-485e-9bb7-ba2641d40834-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f3cf07-299e-485e-9bb7-ba2641d40834-istiod-ca-cert\") pod \"34f3cf07-299e-485e-9bb7-ba2641d40834\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") "
Oct 30 19:18:51.408643 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408485    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-certs\") pod \"34f3cf07-299e-485e-9bb7-ba2641d40834\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") "
Oct 30 19:18:51.408643 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408555    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-kube-api-access-8lht7\" (UniqueName: \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-kube-api-access-8lht7\") pod \"34f3cf07-299e-485e-9bb7-ba2641d40834\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") "
Oct 30 19:18:51.408643 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408602    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-data\") pod \"34f3cf07-299e-485e-9bb7-ba2641d40834\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") "
Oct 30 19:18:51.408643 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408613    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-credential-socket" pod "34f3cf07-299e-485e-9bb7-ba2641d40834" (UID: "34f3cf07-299e-485e-9bb7-ba2641d40834"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.408858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408650    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-token\" (UniqueName: \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-token\") pod \"34f3cf07-299e-485e-9bb7-ba2641d40834\" (UID: \"34f3cf07-299e-485e-9bb7-ba2641d40834\") "
Oct 30 19:18:51.408858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408721    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-certs" pod "34f3cf07-299e-485e-9bb7-ba2641d40834" (UID: "34f3cf07-299e-485e-9bb7-ba2641d40834"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.408858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408809    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f3cf07-299e-485e-9bb7-ba2641d40834-istiod-ca-cert" pod "34f3cf07-299e-485e-9bb7-ba2641d40834" (UID: "34f3cf07-299e-485e-9bb7-ba2641d40834"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:51.409037 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.408940    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-data" pod "34f3cf07-299e-485e-9bb7-ba2641d40834" (UID: "34f3cf07-299e-485e-9bb7-ba2641d40834"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.409229 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.409209    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.409302 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.409235    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.409302 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.409250    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34f3cf07-299e-485e-9bb7-ba2641d40834-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.409302 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.409264    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.409302 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.409277    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.410731 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.410705    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-envoy" pod "34f3cf07-299e-485e-9bb7-ba2641d40834" (UID: "34f3cf07-299e-485e-9bb7-ba2641d40834"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.410816 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.410735    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-kube-api-access-8lht7" pod "34f3cf07-299e-485e-9bb7-ba2641d40834" (UID: "34f3cf07-299e-485e-9bb7-ba2641d40834"). InnerVolumeSpecName "kube-api-access-8lht7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.410816 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.410746    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-podinfo" pod "34f3cf07-299e-485e-9bb7-ba2641d40834" (UID: "34f3cf07-299e-485e-9bb7-ba2641d40834"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:51.410816 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.410786    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-token" pod "34f3cf07-299e-485e-9bb7-ba2641d40834" (UID: "34f3cf07-299e-485e-9bb7-ba2641d40834"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.509882 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.509850    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-credential-socket\") pod \"f04ad0d0-e089-4dce-8735-968a6eae12e5\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") "
Oct 30 19:18:51.510067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.509911    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-kube-api-access-x54lr\" (UniqueName: \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-kube-api-access-x54lr\") pod \"f04ad0d0-e089-4dce-8735-968a6eae12e5\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") "
Oct 30 19:18:51.510067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.509932    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04ad0d0-e089-4dce-8735-968a6eae12e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04ad0d0-e089-4dce-8735-968a6eae12e5-istiod-ca-cert\") pod \"f04ad0d0-e089-4dce-8735-968a6eae12e5\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") "
Oct 30 19:18:51.510067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.509952    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-socket\") pod \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") "
Oct 30 19:18:51.510067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.509973    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-socket\") pod \"f04ad0d0-e089-4dce-8735-968a6eae12e5\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") "
Oct 30 19:18:51.510067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510001    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-data\") pod \"f04ad0d0-e089-4dce-8735-968a6eae12e5\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") "
Oct 30 19:18:51.510067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510034    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istiod-ca-cert\") pod \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") "
Oct 30 19:18:51.510067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510067    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-credential-socket\") pod \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") "
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510102    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-credential-socket" pod "f04ad0d0-e089-4dce-8735-968a6eae12e5" (UID: "f04ad0d0-e089-4dce-8735-968a6eae12e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510118    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-podinfo\") pod \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") "
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510209    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-certs\") pod \"f04ad0d0-e089-4dce-8735-968a6eae12e5\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") "
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510241    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-socket" pod "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" (UID: "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510304    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-certs\") pod \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") "
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510338    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-socket" pod "f04ad0d0-e089-4dce-8735-968a6eae12e5" (UID: "f04ad0d0-e089-4dce-8735-968a6eae12e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510345    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-token\" (UniqueName: \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-token\") pod \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") "
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510378    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-envoy\") pod \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") "
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510402    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-data" pod "f04ad0d0-e089-4dce-8735-968a6eae12e5" (UID: "f04ad0d0-e089-4dce-8735-968a6eae12e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510417    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-credential-socket" pod "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" (UID: "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.510486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510478    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-kube-api-access-vzwmz\" (UniqueName: \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-kube-api-access-vzwmz\") pod \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") "
Oct 30 19:18:51.511102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510540    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istiod-ca-cert" pod "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" (UID: "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:51.511102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510554    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-data\") pod \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\" (UID: \"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d\") "
Oct 30 19:18:51.511102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510563    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-certs" pod "f04ad0d0-e089-4dce-8735-968a6eae12e5" (UID: "f04ad0d0-e089-4dce-8735-968a6eae12e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.511102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510689    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-certs" pod "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" (UID: "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.511102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510789    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-data" pod "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" (UID: "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.511102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510797    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-podinfo\") pod \"f04ad0d0-e089-4dce-8735-968a6eae12e5\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") "
Oct 30 19:18:51.511102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510833    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-envoy\") pod \"f04ad0d0-e089-4dce-8735-968a6eae12e5\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") "
Oct 30 19:18:51.511102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.510882    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-token\" (UniqueName: \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-token\") pod \"f04ad0d0-e089-4dce-8735-968a6eae12e5\" (UID: \"f04ad0d0-e089-4dce-8735-968a6eae12e5\") "
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511370    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511393    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511411    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lht7\" (UniqueName: \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-kube-api-access-8lht7\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511430    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511454    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511468    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511483    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511516    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511532    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511547    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34f3cf07-299e-485e-9bb7-ba2641d40834-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511563    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511578    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.511596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511593    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.512283 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.511939    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04ad0d0-e089-4dce-8735-968a6eae12e5-istiod-ca-cert" pod "f04ad0d0-e089-4dce-8735-968a6eae12e5" (UID: "f04ad0d0-e089-4dce-8735-968a6eae12e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:51.512932 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.512902    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-token" pod "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" (UID: "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.513076 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.513042    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-kube-api-access-x54lr" pod "f04ad0d0-e089-4dce-8735-968a6eae12e5" (UID: "f04ad0d0-e089-4dce-8735-968a6eae12e5"). InnerVolumeSpecName "kube-api-access-x54lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.513199 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.513083    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-envoy" pod "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" (UID: "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.513199 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.513191    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-podinfo" pod "f04ad0d0-e089-4dce-8735-968a6eae12e5" (UID: "f04ad0d0-e089-4dce-8735-968a6eae12e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:51.513320 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.513207    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-podinfo" pod "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" (UID: "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:51.513320 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.513216    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-kube-api-access-vzwmz" pod "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" (UID: "d8a0bd21-4b46-40dc-99f7-fe2c251fb93d"). InnerVolumeSpecName "kube-api-access-vzwmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.514046 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.514025    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-token" pod "f04ad0d0-e089-4dce-8735-968a6eae12e5" (UID: "f04ad0d0-e089-4dce-8735-968a6eae12e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:51.514168 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.514147    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-envoy" pod "f04ad0d0-e089-4dce-8735-968a6eae12e5" (UID: "f04ad0d0-e089-4dce-8735-968a6eae12e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:51.612857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.612781    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.612857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.612813    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.612857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.612824    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwmz\" (UniqueName: \"kubernetes.io/projected/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-kube-api-access-vzwmz\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.612857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.612834    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.612857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.612844    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.612857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.612854    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.612857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.612862    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x54lr\" (UniqueName: \"kubernetes.io/projected/f04ad0d0-e089-4dce-8735-968a6eae12e5-kube-api-access-x54lr\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.613204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.612871    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f04ad0d0-e089-4dce-8735-968a6eae12e5-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:51.613204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:51.612879    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:52.193519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.193462    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"
Oct 30 19:18:52.193519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.193487    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2" event={"ID":"d8a0bd21-4b46-40dc-99f7-fe2c251fb93d","Type":"ContainerDied","Data":"5a5d387a594d575565f150c4002563f668376dfdd691a0fbe4d5e1f3ddd69f7d"}
Oct 30 19:18:52.194035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.193572    4267 scope.go:122] "RemoveContainer" containerID="2cbd31ddda16c23e84ddbf989935cc645f6caf1e6df6e878b0764bff8979398d"
Oct 30 19:18:52.195464 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.195446    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"
Oct 30 19:18:52.195683 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.195472    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb" event={"ID":"f04ad0d0-e089-4dce-8735-968a6eae12e5","Type":"ContainerDied","Data":"f30d9b04ea5913f3973278f9dec2a2496447ec18570bd893e6e7b2d3f333e2ec"}
Oct 30 19:18:52.197089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.197066    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w" event={"ID":"34f3cf07-299e-485e-9bb7-ba2641d40834","Type":"ContainerDied","Data":"5af6a6f380aec2a0179fcdee6a0b7e6ec5450e3686182ff13dc83671a7dab56b"}
Oct 30 19:18:52.197089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.197079    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"
Oct 30 19:18:52.205700 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.205679    4267 scope.go:122] "RemoveContainer" containerID="86626663480cd2f650d2043e2f8f86dc167e523ba5eb866233cf3e74a9383e4e"
Oct 30 19:18:52.216164 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.216138    4267 scope.go:122] "RemoveContainer" containerID="882d8eb23e86bf8429b530cfb89a32bcf8d9f111ca18e3c4d6c96000629fac5c"
Oct 30 19:18:52.359999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.359900    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"]
Oct 30 19:18:52.409257 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.409219    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-d2hd2"]
Oct 30 19:18:52.472987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.472944    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"]
Oct 30 19:18:52.475944 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.475920    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-6htdb"]
Oct 30 19:18:52.601149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.601108    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"]
Oct 30 19:18:52.612823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:52.612754    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ae6a138d-openshift-default-557d46bb4f-rpq6w"]
Oct 30 19:18:53.127312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127277    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"]
Oct 30 19:18:53.127778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127764    4267 cpu_manager.go:414] "containerMap: removing container" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerName="istio-proxy"
Oct 30 19:18:53.127835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127780    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerName="istio-proxy"
Oct 30 19:18:53.127835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127795    4267 cpu_manager.go:414] "containerMap: removing container" podUID="62062c97-8d55-4933-b39e-8961ed4925b9" containerName="payload-processing"
Oct 30 19:18:53.127835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127801    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62062c97-8d55-4933-b39e-8961ed4925b9" containerName="payload-processing"
Oct 30 19:18:53.127835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127817    4267 cpu_manager.go:414] "containerMap: removing container" podUID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" containerName="istio-proxy"
Oct 30 19:18:53.127835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127822    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" containerName="istio-proxy"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127839    4267 cpu_manager.go:414] "containerMap: removing container" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerName="istio-proxy"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127844    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerName="istio-proxy"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127852    4267 cpu_manager.go:414] "containerMap: removing container" podUID="91d78de9-1580-4e7b-a3e3-e0a79f32dcf2" containerName="maas-api"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127857    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d78de9-1580-4e7b-a3e3-e0a79f32dcf2" containerName="maas-api"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127863    4267 cpu_manager.go:414] "containerMap: removing container" podUID="d304bea0-f8b8-43b5-a243-c49592c152ac" containerName="revoke-keys"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127867    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d304bea0-f8b8-43b5-a243-c49592c152ac" containerName="revoke-keys"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127931    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" containerName="istio-proxy"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127937    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="d304bea0-f8b8-43b5-a243-c49592c152ac" containerName="revoke-keys"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127944    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d78de9-1580-4e7b-a3e3-e0a79f32dcf2" containerName="maas-api"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127953    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" containerName="istio-proxy"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127959    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="62062c97-8d55-4933-b39e-8961ed4925b9" containerName="payload-processing"
Oct 30 19:18:53.128014 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.127966    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" containerName="istio-proxy"
Oct 30 19:18:53.133108 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.133087    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.185687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.185643    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:18:53.185910 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.185870    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:18:53.190253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.190226    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"]
Oct 30 19:18:53.222238 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.222202    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f3cf07-299e-485e-9bb7-ba2641d40834" path="/var/lib/kubelet/pods/34f3cf07-299e-485e-9bb7-ba2641d40834/volumes"
Oct 30 19:18:53.222871 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.222854    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a0bd21-4b46-40dc-99f7-fe2c251fb93d" path="/var/lib/kubelet/pods/d8a0bd21-4b46-40dc-99f7-fe2c251fb93d/volumes"
Oct 30 19:18:53.223274 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.223261    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04ad0d0-e089-4dce-8735-968a6eae12e5" path="/var/lib/kubelet/pods/f04ad0d0-e089-4dce-8735-968a6eae12e5/volumes"
Oct 30 19:18:53.230732 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.230701    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686e2376-627dp\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.230894 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.230778    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4bq\" (UniqueName: \"kubernetes.io/projected/c4253ad0-ff62-4ac3-9b21-68644674c4f9-kube-api-access-9r4bq\") pod \"maas-api-revoke-sub-686e2376-627dp\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.230992 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.230963    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686e2376-627dp\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.332475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.332437    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4bq\" (UniqueName: \"kubernetes.io/projected/c4253ad0-ff62-4ac3-9b21-68644674c4f9-kube-api-access-9r4bq\") pod \"maas-api-revoke-sub-686e2376-627dp\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.332790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.332768    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686e2376-627dp\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.332883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.332869    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686e2376-627dp\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.333434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.333408    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686e2376-627dp\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.335229 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.335208    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686e2376-627dp\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.390249 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.390174    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4bq\" (UniqueName: \"kubernetes.io/projected/c4253ad0-ff62-4ac3-9b21-68644674c4f9-kube-api-access-9r4bq\") pod \"maas-api-revoke-sub-686e2376-627dp\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.446348 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.446309    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:53.824717 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:53.824681    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"]
Oct 30 19:18:53.825239 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:18:53.825213    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4253ad0_ff62_4ac3_9b21_68644674c4f9.slice/crio-319e1f7e5deb3d80ecc43d54f1cf4e8c8566409eabf3a7eb974b71a4fd63e292 WatchSource:0}: Error finding container 319e1f7e5deb3d80ecc43d54f1cf4e8c8566409eabf3a7eb974b71a4fd63e292: Status 404 returned error can't find the container with id 319e1f7e5deb3d80ecc43d54f1cf4e8c8566409eabf3a7eb974b71a4fd63e292
Oct 30 19:18:54.217325 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:54.217282    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp" event={"ID":"c4253ad0-ff62-4ac3-9b21-68644674c4f9","Type":"ContainerStarted","Data":"e9068893019945d137c47c50d0684d6d976f33767d8af23d7305a01611966aa6"}
Oct 30 19:18:54.217527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:54.217345    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp" event={"ID":"c4253ad0-ff62-4ac3-9b21-68644674c4f9","Type":"ContainerStarted","Data":"319e1f7e5deb3d80ecc43d54f1cf4e8c8566409eabf3a7eb974b71a4fd63e292"}
Oct 30 19:18:54.332090 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:54.332032    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp" podStartSLOduration=1.332019019 podStartE2EDuration="1.332019019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:54.330631969 +0000 UTC m=+2425.218549059" watchObservedRunningTime="2026-10-30 19:18:54.332019019 +0000 UTC m=+2425.219936108"
Oct 30 19:18:55.224114 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:55.224080    4267 generic.go:356] "Generic (PLEG): container finished" podID="c4253ad0-ff62-4ac3-9b21-68644674c4f9" containerID="e9068893019945d137c47c50d0684d6d976f33767d8af23d7305a01611966aa6" exitCode=0
Oct 30 19:18:55.224307 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:55.224180    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp" event={"ID":"c4253ad0-ff62-4ac3-9b21-68644674c4f9","Type":"ContainerDied","Data":"e9068893019945d137c47c50d0684d6d976f33767d8af23d7305a01611966aa6"}
Oct 30 19:18:56.383121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.383089    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:18:56.466377 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.466335    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4253ad0-ff62-4ac3-9b21-68644674c4f9-kube-api-access-9r4bq\" (UniqueName: \"kubernetes.io/projected/c4253ad0-ff62-4ac3-9b21-68644674c4f9-kube-api-access-9r4bq\") pod \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") "
Oct 30 19:18:56.466621 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.466451    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-xks-ca\") pod \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") "
Oct 30 19:18:56.466621 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.466525    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-openshift-ca\") pod \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\" (UID: \"c4253ad0-ff62-4ac3-9b21-68644674c4f9\") "
Oct 30 19:18:56.466987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.466955    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-openshift-ca" pod "c4253ad0-ff62-4ac3-9b21-68644674c4f9" (UID: "c4253ad0-ff62-4ac3-9b21-68644674c4f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:56.468701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.468664    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4253ad0-ff62-4ac3-9b21-68644674c4f9-kube-api-access-9r4bq" pod "c4253ad0-ff62-4ac3-9b21-68644674c4f9" (UID: "c4253ad0-ff62-4ac3-9b21-68644674c4f9"). InnerVolumeSpecName "kube-api-access-9r4bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:56.468983 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.468966    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-xks-ca" pod "c4253ad0-ff62-4ac3-9b21-68644674c4f9" (UID: "c4253ad0-ff62-4ac3-9b21-68644674c4f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:56.568323 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.568233    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:56.568323 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.568267    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4253ad0-ff62-4ac3-9b21-68644674c4f9-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:56.568323 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.568279    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r4bq\" (UniqueName: \"kubernetes.io/projected/c4253ad0-ff62-4ac3-9b21-68644674c4f9-kube-api-access-9r4bq\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:18:56.871817 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.871730    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"]
Oct 30 19:18:56.872007 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:56.871969    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" podUID="5117062a-c82a-41d6-8286-526d69c1504c" containerName="istio-proxy" containerID="cri-o://41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b" gracePeriod=30
Oct 30 19:18:57.240726 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:57.240677    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp" event={"ID":"c4253ad0-ff62-4ac3-9b21-68644674c4f9","Type":"ContainerDied","Data":"319e1f7e5deb3d80ecc43d54f1cf4e8c8566409eabf3a7eb974b71a4fd63e292"}
Oct 30 19:18:57.240966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:57.240752    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="319e1f7e5deb3d80ecc43d54f1cf4e8c8566409eabf3a7eb974b71a4fd63e292"
Oct 30 19:18:57.240966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:18:57.240839    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"
Oct 30 19:19:02.176663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.176626    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 19:19:02.282027 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.281991    4267 generic.go:356] "Generic (PLEG): container finished" podID="5117062a-c82a-41d6-8286-526d69c1504c" containerID="41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b" exitCode=0
Oct 30 19:19:02.282322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.282052    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"
Oct 30 19:19:02.282415 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.282044    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" event={"ID":"5117062a-c82a-41d6-8286-526d69c1504c","Type":"ContainerDied","Data":"41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b"}
Oct 30 19:19:02.282511 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.282467    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt" event={"ID":"5117062a-c82a-41d6-8286-526d69c1504c","Type":"ContainerDied","Data":"99bcbe9f63b78b1bf48afd0cb6a8779875c9c8d48f621cfc9798208a70341a66"}
Oct 30 19:19:02.282574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.282532    4267 scope.go:122] "RemoveContainer" containerID="41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b"
Oct 30 19:19:02.295915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.295887    4267 scope.go:122] "RemoveContainer" containerID="41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b"
Oct 30 19:19:02.296552 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:19:02.296478    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b\": container with ID starting with 41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b not found: ID does not exist" containerID="41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b"
Oct 30 19:19:02.296667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.296567    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b"} err="failed to get container status \"41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b\": rpc error: code = NotFound desc = could not find container \"41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b\": container with ID starting with 41f44ff22ceb2ae92d2d6098de201f2098454667718d13e8db68e6fa14287e1b not found: ID does not exist"
Oct 30 19:19:02.346526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.345597    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-credential-socket\") pod \"5117062a-c82a-41d6-8286-526d69c1504c\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") "
Oct 30 19:19:02.346526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.345661    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-istio-token\" (UniqueName: \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-istio-token\") pod \"5117062a-c82a-41d6-8286-526d69c1504c\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") "
Oct 30 19:19:02.346526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.345697    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5117062a-c82a-41d6-8286-526d69c1504c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5117062a-c82a-41d6-8286-526d69c1504c-istio-podinfo\") pod \"5117062a-c82a-41d6-8286-526d69c1504c\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") "
Oct 30 19:19:02.346526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.345752    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-envoy\") pod \"5117062a-c82a-41d6-8286-526d69c1504c\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") "
Oct 30 19:19:02.346526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.345811    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5117062a-c82a-41d6-8286-526d69c1504c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5117062a-c82a-41d6-8286-526d69c1504c-istiod-ca-cert\") pod \"5117062a-c82a-41d6-8286-526d69c1504c\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") "
Oct 30 19:19:02.346526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.345853    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-socket\") pod \"5117062a-c82a-41d6-8286-526d69c1504c\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") "
Oct 30 19:19:02.346526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.345903    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-certs\") pod \"5117062a-c82a-41d6-8286-526d69c1504c\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") "
Oct 30 19:19:02.346526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.345936    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-data\") pod \"5117062a-c82a-41d6-8286-526d69c1504c\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") "
Oct 30 19:19:02.346526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.346048    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-kube-api-access-8h2v5\" (UniqueName: \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-kube-api-access-8h2v5\") pod \"5117062a-c82a-41d6-8286-526d69c1504c\" (UID: \"5117062a-c82a-41d6-8286-526d69c1504c\") "
Oct 30 19:19:02.347153 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.346992    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-credential-socket" pod "5117062a-c82a-41d6-8286-526d69c1504c" (UID: "5117062a-c82a-41d6-8286-526d69c1504c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:02.350056 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.349814    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-istio-token" pod "5117062a-c82a-41d6-8286-526d69c1504c" (UID: "5117062a-c82a-41d6-8286-526d69c1504c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:02.350056 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.350001    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-certs" pod "5117062a-c82a-41d6-8286-526d69c1504c" (UID: "5117062a-c82a-41d6-8286-526d69c1504c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:02.350439 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.350242    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-socket" pod "5117062a-c82a-41d6-8286-526d69c1504c" (UID: "5117062a-c82a-41d6-8286-526d69c1504c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:02.350439 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.350279    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-data" pod "5117062a-c82a-41d6-8286-526d69c1504c" (UID: "5117062a-c82a-41d6-8286-526d69c1504c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:02.350439 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.350407    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5117062a-c82a-41d6-8286-526d69c1504c-istiod-ca-cert" pod "5117062a-c82a-41d6-8286-526d69c1504c" (UID: "5117062a-c82a-41d6-8286-526d69c1504c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:02.355312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.355255    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-envoy" pod "5117062a-c82a-41d6-8286-526d69c1504c" (UID: "5117062a-c82a-41d6-8286-526d69c1504c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:02.355312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.355280    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-kube-api-access-8h2v5" pod "5117062a-c82a-41d6-8286-526d69c1504c" (UID: "5117062a-c82a-41d6-8286-526d69c1504c"). InnerVolumeSpecName "kube-api-access-8h2v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:02.355475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.355409    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5117062a-c82a-41d6-8286-526d69c1504c-istio-podinfo" pod "5117062a-c82a-41d6-8286-526d69c1504c" (UID: "5117062a-c82a-41d6-8286-526d69c1504c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:19:02.447695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.447645    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:02.447695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.447683    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:02.447695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.447698    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:02.447933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.447711    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h2v5\" (UniqueName: \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-kube-api-access-8h2v5\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:02.447933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.447726    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:02.447933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.447737    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5117062a-c82a-41d6-8286-526d69c1504c-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:02.447933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.447748    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5117062a-c82a-41d6-8286-526d69c1504c-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:02.447933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.447759    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5117062a-c82a-41d6-8286-526d69c1504c-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:02.447933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.447771    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5117062a-c82a-41d6-8286-526d69c1504c-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:02.742579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.741545    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"]
Oct 30 19:19:02.754528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:02.751591    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bh5zt"]
Oct 30 19:19:03.235089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:03.235036    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5117062a-c82a-41d6-8286-526d69c1504c" path="/var/lib/kubelet/pods/5117062a-c82a-41d6-8286-526d69c1504c/volumes"
Oct 30 19:19:04.064514 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.063873    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"]
Oct 30 19:19:04.075516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.068669    4267 cpu_manager.go:414] "containerMap: removing container" podUID="5117062a-c82a-41d6-8286-526d69c1504c" containerName="istio-proxy"
Oct 30 19:19:04.075516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.068712    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5117062a-c82a-41d6-8286-526d69c1504c" containerName="istio-proxy"
Oct 30 19:19:04.075516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.068742    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c4253ad0-ff62-4ac3-9b21-68644674c4f9" containerName="revoke-keys"
Oct 30 19:19:04.075516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.068752    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4253ad0-ff62-4ac3-9b21-68644674c4f9" containerName="revoke-keys"
Oct 30 19:19:04.075516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.068940    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="5117062a-c82a-41d6-8286-526d69c1504c" containerName="istio-proxy"
Oct 30 19:19:04.075516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.068956    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4253ad0-ff62-4ac3-9b21-68644674c4f9" containerName="revoke-keys"
Oct 30 19:19:04.075516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.074533    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.088585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.088215    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.088585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.088277    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf65r\" (UniqueName: \"kubernetes.io/projected/9ba5ad96-fce2-445b-8763-8093185d58f5-kube-api-access-xf65r\") pod \"maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.088585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.088405    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.128555 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.124194    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"]
Oct 30 19:19:04.156619 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.155745    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:04.156619 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.156412    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:19:04.196864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.196826    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.197240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.197215    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.197386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.197370    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf65r\" (UniqueName: \"kubernetes.io/projected/9ba5ad96-fce2-445b-8763-8093185d58f5-kube-api-access-xf65r\") pod \"maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.199016 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.198989    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.253518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.228016    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.277533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.274574    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf65r\" (UniqueName: \"kubernetes.io/projected/9ba5ad96-fce2-445b-8763-8093185d58f5-kube-api-access-xf65r\") pod \"maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.414504 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.414377    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:04.836669 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:19:04.836630    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba5ad96_fce2_445b_8763_8093185d58f5.slice/crio-446cfe89ffd57380f14d9de4b688a2ca7c2acc7bb6a7547bbf0ee7499cfd3858 WatchSource:0}: Error finding container 446cfe89ffd57380f14d9de4b688a2ca7c2acc7bb6a7547bbf0ee7499cfd3858: Status 404 returned error can't find the container with id 446cfe89ffd57380f14d9de4b688a2ca7c2acc7bb6a7547bbf0ee7499cfd3858
Oct 30 19:19:04.842603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:04.842576    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"]
Oct 30 19:19:05.353278 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:05.353240    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7" event={"ID":"9ba5ad96-fce2-445b-8763-8093185d58f5","Type":"ContainerStarted","Data":"2870432d14c5e8e9d4d4efaf24af1ccab50b3de821787262f340f4118bb6d8c9"}
Oct 30 19:19:05.353667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:05.353294    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7" event={"ID":"9ba5ad96-fce2-445b-8763-8093185d58f5","Type":"ContainerStarted","Data":"446cfe89ffd57380f14d9de4b688a2ca7c2acc7bb6a7547bbf0ee7499cfd3858"}
Oct 30 19:19:05.468116 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:05.468054    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7" podStartSLOduration=2.468039748 podStartE2EDuration="2.468039748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:05.465611627 +0000 UTC m=+2436.353528713" watchObservedRunningTime="2026-10-30 19:19:05.468039748 +0000 UTC m=+2436.355956816"
Oct 30 19:19:06.361694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:06.361658    4267 generic.go:356] "Generic (PLEG): container finished" podID="9ba5ad96-fce2-445b-8763-8093185d58f5" containerID="2870432d14c5e8e9d4d4efaf24af1ccab50b3de821787262f340f4118bb6d8c9" exitCode=0
Oct 30 19:19:06.362186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:06.361744    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7" event={"ID":"9ba5ad96-fce2-445b-8763-8093185d58f5","Type":"ContainerDied","Data":"2870432d14c5e8e9d4d4efaf24af1ccab50b3de821787262f340f4118bb6d8c9"}
Oct 30 19:19:07.524774 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:07.524745    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:07.542038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:07.541999    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-openshift-ca\") pod \"9ba5ad96-fce2-445b-8763-8093185d58f5\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") "
Oct 30 19:19:07.542223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:07.542073    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-xks-ca\") pod \"9ba5ad96-fce2-445b-8763-8093185d58f5\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") "
Oct 30 19:19:07.542223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:07.542195    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba5ad96-fce2-445b-8763-8093185d58f5-kube-api-access-xf65r\" (UniqueName: \"kubernetes.io/projected/9ba5ad96-fce2-445b-8763-8093185d58f5-kube-api-access-xf65r\") pod \"9ba5ad96-fce2-445b-8763-8093185d58f5\" (UID: \"9ba5ad96-fce2-445b-8763-8093185d58f5\") "
Oct 30 19:19:07.542374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:07.542350    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-openshift-ca" pod "9ba5ad96-fce2-445b-8763-8093185d58f5" (UID: "9ba5ad96-fce2-445b-8763-8093185d58f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:07.542600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:07.542581    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:07.544178 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:07.544150    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-xks-ca" pod "9ba5ad96-fce2-445b-8763-8093185d58f5" (UID: "9ba5ad96-fce2-445b-8763-8093185d58f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:07.544258 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:07.544223    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba5ad96-fce2-445b-8763-8093185d58f5-kube-api-access-xf65r" pod "9ba5ad96-fce2-445b-8763-8093185d58f5" (UID: "9ba5ad96-fce2-445b-8763-8093185d58f5"). InnerVolumeSpecName "kube-api-access-xf65r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:07.643459 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:07.643376    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf65r\" (UniqueName: \"kubernetes.io/projected/9ba5ad96-fce2-445b-8763-8093185d58f5-kube-api-access-xf65r\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:07.643459 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:07.643403    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ba5ad96-fce2-445b-8763-8093185d58f5-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:08.378016 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:08.377981    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7" event={"ID":"9ba5ad96-fce2-445b-8763-8093185d58f5","Type":"ContainerDied","Data":"446cfe89ffd57380f14d9de4b688a2ca7c2acc7bb6a7547bbf0ee7499cfd3858"}
Oct 30 19:19:08.378203 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:08.378018    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"
Oct 30 19:19:08.378203 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:08.378030    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="446cfe89ffd57380f14d9de4b688a2ca7c2acc7bb6a7547bbf0ee7499cfd3858"
Oct 30 19:19:08.742047 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:08.742005    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"]
Oct 30 19:19:08.758653 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:08.758617    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"]
Oct 30 19:19:12.001364 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:12.001323    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"]
Oct 30 19:19:12.001833 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:12.001570    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy" containerID="cri-o://1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3" gracePeriod=30
Oct 30 19:19:13.294915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.294879    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"]
Oct 30 19:19:13.295419 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.295079    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc" podUID="ec566702-3dbf-4722-8a8b-a54f08cda9e9" containerName="maas-api" containerID="cri-o://5dcaf96053e7a2334efba9e554b2c5614bae97fb7c90638daccdc41c99efad16" gracePeriod=30
Oct 30 19:19:13.413738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.413707    4267 generic.go:356] "Generic (PLEG): container finished" podID="ec566702-3dbf-4722-8a8b-a54f08cda9e9" containerID="5dcaf96053e7a2334efba9e554b2c5614bae97fb7c90638daccdc41c99efad16" exitCode=0
Oct 30 19:19:13.413911 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.413783    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc" event={"ID":"ec566702-3dbf-4722-8a8b-a54f08cda9e9","Type":"ContainerDied","Data":"5dcaf96053e7a2334efba9e554b2c5614bae97fb7c90638daccdc41c99efad16"}
Oct 30 19:19:13.592977 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.592947    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 19:19:13.715009 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.714958    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-metrics-certs\") pod \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") "
Oct 30 19:19:13.715188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.715035    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec566702-3dbf-4722-8a8b-a54f08cda9e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec566702-3dbf-4722-8a8b-a54f08cda9e9-openshift-service-ca\") pod \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") "
Oct 30 19:19:13.715188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.715108    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-maas-api-tls\") pod \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") "
Oct 30 19:19:13.715338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.715196    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec566702-3dbf-4722-8a8b-a54f08cda9e9-kube-api-access-8mp9g\" (UniqueName: \"kubernetes.io/projected/ec566702-3dbf-4722-8a8b-a54f08cda9e9-kube-api-access-8mp9g\") pod \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\" (UID: \"ec566702-3dbf-4722-8a8b-a54f08cda9e9\") "
Oct 30 19:19:13.715744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.715414    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec566702-3dbf-4722-8a8b-a54f08cda9e9-openshift-service-ca" pod "ec566702-3dbf-4722-8a8b-a54f08cda9e9" (UID: "ec566702-3dbf-4722-8a8b-a54f08cda9e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:13.715744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.715696    4267 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec566702-3dbf-4722-8a8b-a54f08cda9e9-openshift-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:13.717664 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.717636    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-metrics-certs" pod "ec566702-3dbf-4722-8a8b-a54f08cda9e9" (UID: "ec566702-3dbf-4722-8a8b-a54f08cda9e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:13.717893 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.717866    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-maas-api-tls" pod "ec566702-3dbf-4722-8a8b-a54f08cda9e9" (UID: "ec566702-3dbf-4722-8a8b-a54f08cda9e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:13.718205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.718181    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec566702-3dbf-4722-8a8b-a54f08cda9e9-kube-api-access-8mp9g" pod "ec566702-3dbf-4722-8a8b-a54f08cda9e9" (UID: "ec566702-3dbf-4722-8a8b-a54f08cda9e9"). InnerVolumeSpecName "kube-api-access-8mp9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:13.817205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.817164    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-maas-api-tls\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:13.817205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.817201    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mp9g\" (UniqueName: \"kubernetes.io/projected/ec566702-3dbf-4722-8a8b-a54f08cda9e9-kube-api-access-8mp9g\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:13.817205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:13.817211    4267 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec566702-3dbf-4722-8a8b-a54f08cda9e9-metrics-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:14.421395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:14.421354    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc" event={"ID":"ec566702-3dbf-4722-8a8b-a54f08cda9e9","Type":"ContainerDied","Data":"9712d67c92935de651b56aaa16271437e20ccbcc03043cd307c5b7f375ac42a8"}
Oct 30 19:19:14.421908 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:14.421369    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"
Oct 30 19:19:14.421908 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:14.421456    4267 scope.go:122] "RemoveContainer" containerID="5dcaf96053e7a2334efba9e554b2c5614bae97fb7c90638daccdc41c99efad16"
Oct 30 19:19:14.593329 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:14.593295    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"]
Oct 30 19:19:14.636645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:14.636614    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-1290a6-56db97fc69-m86pc"]
Oct 30 19:19:15.220270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:15.220232    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec566702-3dbf-4722-8a8b-a54f08cda9e9" path="/var/lib/kubelet/pods/ec566702-3dbf-4722-8a8b-a54f08cda9e9/volumes"
Oct 30 19:19:15.529215 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:15.529108    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"]
Oct 30 19:19:15.576406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:15.576366    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"]
Oct 30 19:19:15.579203 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:15.579175    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1290a6-t8j44"]
Oct 30 19:19:17.226762 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.226725    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef3a653-91ad-4038-af04-eba703cc0644" path="/var/lib/kubelet/pods/0ef3a653-91ad-4038-af04-eba703cc0644/volumes"
Oct 30 19:19:17.324132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.324052    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 19:19:17.363864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.363825    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-data\") pod \"64927929-46af-4325-80de-45c8eca88e5d\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") "
Oct 30 19:19:17.364068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.363883    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-certs\") pod \"64927929-46af-4325-80de-45c8eca88e5d\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") "
Oct 30 19:19:17.364068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.363920    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64927929-46af-4325-80de-45c8eca88e5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64927929-46af-4325-80de-45c8eca88e5d-istiod-ca-cert\") pod \"64927929-46af-4325-80de-45c8eca88e5d\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") "
Oct 30 19:19:17.364068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.363983    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-istio-token\" (UniqueName: \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-istio-token\") pod \"64927929-46af-4325-80de-45c8eca88e5d\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") "
Oct 30 19:19:17.364253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364105    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-envoy\") pod \"64927929-46af-4325-80de-45c8eca88e5d\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") "
Oct 30 19:19:17.364253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364149    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-credential-socket\") pod \"64927929-46af-4325-80de-45c8eca88e5d\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") "
Oct 30 19:19:17.364253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364149    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-data" pod "64927929-46af-4325-80de-45c8eca88e5d" (UID: "64927929-46af-4325-80de-45c8eca88e5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:17.364253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364167    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-certs" pod "64927929-46af-4325-80de-45c8eca88e5d" (UID: "64927929-46af-4325-80de-45c8eca88e5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:17.364253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364181    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-kube-api-access-wgsnl\" (UniqueName: \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-kube-api-access-wgsnl\") pod \"64927929-46af-4325-80de-45c8eca88e5d\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") "
Oct 30 19:19:17.364427 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364291    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64927929-46af-4325-80de-45c8eca88e5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64927929-46af-4325-80de-45c8eca88e5d-istio-podinfo\") pod \"64927929-46af-4325-80de-45c8eca88e5d\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") "
Oct 30 19:19:17.364427 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364316    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64927929-46af-4325-80de-45c8eca88e5d-istiod-ca-cert" pod "64927929-46af-4325-80de-45c8eca88e5d" (UID: "64927929-46af-4325-80de-45c8eca88e5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:17.364427 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364340    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-socket\") pod \"64927929-46af-4325-80de-45c8eca88e5d\" (UID: \"64927929-46af-4325-80de-45c8eca88e5d\") "
Oct 30 19:19:17.364717 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364692    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-socket" pod "64927929-46af-4325-80de-45c8eca88e5d" (UID: "64927929-46af-4325-80de-45c8eca88e5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:17.364834 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364806    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-credential-socket" pod "64927929-46af-4325-80de-45c8eca88e5d" (UID: "64927929-46af-4325-80de-45c8eca88e5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:17.365007 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.364985    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64927929-46af-4325-80de-45c8eca88e5d-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:17.365153 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.365137    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:17.365276 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.365255    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:17.365371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.365278    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:17.365371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.365293    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:17.366803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.366780    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-kube-api-access-wgsnl" pod "64927929-46af-4325-80de-45c8eca88e5d" (UID: "64927929-46af-4325-80de-45c8eca88e5d"). InnerVolumeSpecName "kube-api-access-wgsnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:17.366907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.366859    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-istio-token" pod "64927929-46af-4325-80de-45c8eca88e5d" (UID: "64927929-46af-4325-80de-45c8eca88e5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:17.366964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.366899    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64927929-46af-4325-80de-45c8eca88e5d-istio-podinfo" pod "64927929-46af-4325-80de-45c8eca88e5d" (UID: "64927929-46af-4325-80de-45c8eca88e5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:19:17.367139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.367120    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-envoy" pod "64927929-46af-4325-80de-45c8eca88e5d" (UID: "64927929-46af-4325-80de-45c8eca88e5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:17.451529 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.451459    4267 generic.go:356] "Generic (PLEG): container finished" podID="64927929-46af-4325-80de-45c8eca88e5d" containerID="1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3" exitCode=0
Oct 30 19:19:17.451712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.451541    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" event={"ID":"64927929-46af-4325-80de-45c8eca88e5d","Type":"ContainerDied","Data":"1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3"}
Oct 30 19:19:17.451712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.451556    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"
Oct 30 19:19:17.451712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.451605    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq" event={"ID":"64927929-46af-4325-80de-45c8eca88e5d","Type":"ContainerDied","Data":"e8c4759ffd050d677e6b287035bfe78b30d8a3dfccfc1b7c466e20e0ddc1542c"}
Oct 30 19:19:17.451712 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.451629    4267 scope.go:122] "RemoveContainer" containerID="1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3"
Oct 30 19:19:17.462062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.462037    4267 scope.go:122] "RemoveContainer" containerID="944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f"
Oct 30 19:19:17.466802 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.466779    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:17.466890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.466803    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64927929-46af-4325-80de-45c8eca88e5d-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:17.466890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.466821    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgsnl\" (UniqueName: \"kubernetes.io/projected/64927929-46af-4325-80de-45c8eca88e5d-kube-api-access-wgsnl\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:17.466890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.466831    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64927929-46af-4325-80de-45c8eca88e5d-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:17.471446 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.471420    4267 scope.go:122] "RemoveContainer" containerID="1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3"
Oct 30 19:19:17.471797 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:19:17.471773    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3\": container with ID starting with 1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3 not found: ID does not exist" containerID="1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3"
Oct 30 19:19:17.471886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.471804    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3"} err="failed to get container status \"1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3\": rpc error: code = NotFound desc = could not find container \"1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3\": container with ID starting with 1f91f6baa45b2de1ec0c41ceaaacca0ae0152162c76510e6cc194db728b155a3 not found: ID does not exist"
Oct 30 19:19:17.471886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.471826    4267 scope.go:122] "RemoveContainer" containerID="944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f"
Oct 30 19:19:17.472076 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:19:17.472056    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f\": container with ID starting with 944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f not found: ID does not exist" containerID="944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f"
Oct 30 19:19:17.472114 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.472081    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f"} err="failed to get container status \"944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f\": rpc error: code = NotFound desc = could not find container \"944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f\": container with ID starting with 944a18a4b7d152a0b2c7a7f38721c0f583f0c5123a8c7d3670280add0e72381f not found: ID does not exist"
Oct 30 19:19:17.605092 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.605014    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"]
Oct 30 19:19:17.607870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:17.607841    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-bmzwq"]
Oct 30 19:19:19.221606 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:19.221571    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64927929-46af-4325-80de-45c8eca88e5d" path="/var/lib/kubelet/pods/64927929-46af-4325-80de-45c8eca88e5d/volumes"
Oct 30 19:19:20.158326 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:19:20.158289    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:19:20.530342 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:20.530295    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj" podUID="96769b37-30bd-4d6c-8035-39bffcbc1f34" containerName="payload-processing" containerID="cri-o://cde569a9a9062ebf30f597444247435bef27b136d9c148ed33b13c5816ceb5d7" gracePeriod=25
Oct 30 19:19:22.341737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.341703    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"]
Oct 30 19:19:22.342295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.341909    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy" containerID="cri-o://aca9ce4380bab96d9ef5d070ef64c55a1aaa0d6759f91c9ff31e77b7400d857c" gracePeriod=30
Oct 30 19:19:22.505072 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.505023    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"]
Oct 30 19:19:22.506095 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506064    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ec566702-3dbf-4722-8a8b-a54f08cda9e9" containerName="maas-api"
Oct 30 19:19:22.506294 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506269    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec566702-3dbf-4722-8a8b-a54f08cda9e9" containerName="maas-api"
Oct 30 19:19:22.506413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506306    4267 cpu_manager.go:414] "containerMap: removing container" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy"
Oct 30 19:19:22.506413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506314    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy"
Oct 30 19:19:22.506413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506331    4267 cpu_manager.go:414] "containerMap: removing container" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy"
Oct 30 19:19:22.506413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506338    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy"
Oct 30 19:19:22.506413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506365    4267 cpu_manager.go:414] "containerMap: removing container" podUID="9ba5ad96-fce2-445b-8763-8093185d58f5" containerName="revoke-keys"
Oct 30 19:19:22.506413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506373    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba5ad96-fce2-445b-8763-8093185d58f5" containerName="revoke-keys"
Oct 30 19:19:22.506630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506507    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba5ad96-fce2-445b-8763-8093185d58f5" containerName="revoke-keys"
Oct 30 19:19:22.506630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506527    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec566702-3dbf-4722-8a8b-a54f08cda9e9" containerName="maas-api"
Oct 30 19:19:22.506630 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.506538    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy"
Oct 30 19:19:22.510273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.510252    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.589969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.589933    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"]
Oct 30 19:19:22.626052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.625962    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.626335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.626307    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.626486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.626368    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.626486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.626406    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.626486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.626426    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.626687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.626505    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.626687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.626608    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.626687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.626679    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2nsj\" (UniqueName: \"kubernetes.io/projected/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-kube-api-access-k2nsj\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.626842 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.626733    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.727593 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.727562    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.727797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.727616    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.727797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.727734    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.727797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.727774    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.727980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.727805    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.727980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.727829    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.727980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.727914    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.728139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.727977    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.728139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.728006    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2nsj\" (UniqueName: \"kubernetes.io/projected/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-kube-api-access-k2nsj\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.728139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.728036    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.728139 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.728090    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.728350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.728265    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.728595 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.728573    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.728752 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.728735    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.729983 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.729965    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.730287 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.730271    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.784323 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.784295    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.784451 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.784430    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2nsj\" (UniqueName: \"kubernetes.io/projected/b4b5cee2-4ba7-4d14-bde2-dd266f4706f6-kube-api-access-k2nsj\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8cp28\" (UID: \"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:22.825955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:22.825929    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:23.250999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.250863    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"]
Oct 30 19:19:23.253604 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:19:23.253559    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b5cee2_4ba7_4d14_bde2_dd266f4706f6.slice/crio-475d6068d78efad80b5868b4eca5e4ccb0d1bcd9274abe43996c9d226cdd645e WatchSource:0}: Error finding container 475d6068d78efad80b5868b4eca5e4ccb0d1bcd9274abe43996c9d226cdd645e: Status 404 returned error can't find the container with id 475d6068d78efad80b5868b4eca5e4ccb0d1bcd9274abe43996c9d226cdd645e
Oct 30 19:19:23.256395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.256359    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:23.256516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.256432    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:23.256516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.256465    4267 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:23.505477 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.505387    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" event={"ID":"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6","Type":"ContainerStarted","Data":"87c846f54de49ae2a2116e32e22f8f8759fd4b8e80601c1755ccd18db4d17474"}
Oct 30 19:19:23.505477 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.505453    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" event={"ID":"b4b5cee2-4ba7-4d14-bde2-dd266f4706f6","Type":"ContainerStarted","Data":"475d6068d78efad80b5868b4eca5e4ccb0d1bcd9274abe43996c9d226cdd645e"}
Oct 30 19:19:23.632960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.632901    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podStartSLOduration=1.6328876810000001 podStartE2EDuration="1.632887681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:23.630057884 +0000 UTC m=+2454.517974976" watchObservedRunningTime="2026-10-30 19:19:23.632887681 +0000 UTC m=+2454.520804771"
Oct 30 19:19:23.769776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.769680    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main" containerID="cri-o://4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f" gracePeriod=45
Oct 30 19:19:23.778661 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.778620    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main" containerID="cri-o://4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810" gracePeriod=45
Oct 30 19:19:23.854964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.844252    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:23.870810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.870780    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"]
Oct 30 19:19:23.871015 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:23.870993    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy" containerID="cri-o://93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f" gracePeriod=30
Oct 30 19:19:24.243560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.234764    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:19:24.246754 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.246729    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:19:24.364308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364228    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-dshm\") pod \"78bb5520-dfd6-49c9-8506-0922401452bd\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") "
Oct 30 19:19:24.364308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364274    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-model-cache\") pod \"02b3f14f-34ea-4635-9e93-d500debd10d9\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") "
Oct 30 19:19:24.364572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364314    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-tmp-dir\") pod \"78bb5520-dfd6-49c9-8506-0922401452bd\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") "
Oct 30 19:19:24.364572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364391    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-home\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-home\") pod \"02b3f14f-34ea-4635-9e93-d500debd10d9\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") "
Oct 30 19:19:24.364572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364419    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-home\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-home\") pod \"78bb5520-dfd6-49c9-8506-0922401452bd\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") "
Oct 30 19:19:24.364572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364451    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78bb5520-dfd6-49c9-8506-0922401452bd-kube-api-access-n5c5k\" (UniqueName: \"kubernetes.io/projected/78bb5520-dfd6-49c9-8506-0922401452bd-kube-api-access-n5c5k\") pod \"78bb5520-dfd6-49c9-8506-0922401452bd\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") "
Oct 30 19:19:24.364572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364483    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b3f14f-34ea-4635-9e93-d500debd10d9-kube-api-access-2w95f\" (UniqueName: \"kubernetes.io/projected/02b3f14f-34ea-4635-9e93-d500debd10d9-kube-api-access-2w95f\") pod \"02b3f14f-34ea-4635-9e93-d500debd10d9\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") "
Oct 30 19:19:24.364853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364588    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-model-cache" pod "02b3f14f-34ea-4635-9e93-d500debd10d9" (UID: "02b3f14f-34ea-4635-9e93-d500debd10d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.364853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364632    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b3f14f-34ea-4635-9e93-d500debd10d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/02b3f14f-34ea-4635-9e93-d500debd10d9-tls-certs\") pod \"02b3f14f-34ea-4635-9e93-d500debd10d9\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") "
Oct 30 19:19:24.364853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364680    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-tmp-dir\") pod \"02b3f14f-34ea-4635-9e93-d500debd10d9\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") "
Oct 30 19:19:24.364853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364734    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78bb5520-dfd6-49c9-8506-0922401452bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/78bb5520-dfd6-49c9-8506-0922401452bd-tls-certs\") pod \"78bb5520-dfd6-49c9-8506-0922401452bd\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") "
Oct 30 19:19:24.364853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364786    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-model-cache\") pod \"78bb5520-dfd6-49c9-8506-0922401452bd\" (UID: \"78bb5520-dfd6-49c9-8506-0922401452bd\") "
Oct 30 19:19:24.364853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364789    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-home" pod "02b3f14f-34ea-4635-9e93-d500debd10d9" (UID: "02b3f14f-34ea-4635-9e93-d500debd10d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.364853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.364832    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-dshm\") pod \"02b3f14f-34ea-4635-9e93-d500debd10d9\" (UID: \"02b3f14f-34ea-4635-9e93-d500debd10d9\") "
Oct 30 19:19:24.365191 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.365104    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-home" pod "78bb5520-dfd6-49c9-8506-0922401452bd" (UID: "78bb5520-dfd6-49c9-8506-0922401452bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.365317 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.365299    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.365390 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.365318    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.365390 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.365328    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.365519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.365437    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-tmp-dir" pod "02b3f14f-34ea-4635-9e93-d500debd10d9" (UID: "02b3f14f-34ea-4635-9e93-d500debd10d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.365519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.365451    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-tmp-dir" pod "78bb5520-dfd6-49c9-8506-0922401452bd" (UID: "78bb5520-dfd6-49c9-8506-0922401452bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.365763 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.365738    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-model-cache" pod "78bb5520-dfd6-49c9-8506-0922401452bd" (UID: "78bb5520-dfd6-49c9-8506-0922401452bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.368131 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.368109    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-dshm" pod "02b3f14f-34ea-4635-9e93-d500debd10d9" (UID: "02b3f14f-34ea-4635-9e93-d500debd10d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.376928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.376888    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78bb5520-dfd6-49c9-8506-0922401452bd-tls-certs" pod "78bb5520-dfd6-49c9-8506-0922401452bd" (UID: "78bb5520-dfd6-49c9-8506-0922401452bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:24.376928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.376897    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bb5520-dfd6-49c9-8506-0922401452bd-kube-api-access-n5c5k" pod "78bb5520-dfd6-49c9-8506-0922401452bd" (UID: "78bb5520-dfd6-49c9-8506-0922401452bd"). InnerVolumeSpecName "kube-api-access-n5c5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:24.376928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.376913    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b3f14f-34ea-4635-9e93-d500debd10d9-tls-certs" pod "02b3f14f-34ea-4635-9e93-d500debd10d9" (UID: "02b3f14f-34ea-4635-9e93-d500debd10d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:24.376928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.376891    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-dshm" pod "78bb5520-dfd6-49c9-8506-0922401452bd" (UID: "78bb5520-dfd6-49c9-8506-0922401452bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:24.378109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.378082    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b3f14f-34ea-4635-9e93-d500debd10d9-kube-api-access-2w95f" pod "02b3f14f-34ea-4635-9e93-d500debd10d9" (UID: "02b3f14f-34ea-4635-9e93-d500debd10d9"). InnerVolumeSpecName "kube-api-access-2w95f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:24.448662 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.448624    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"]
Oct 30 19:19:24.449078 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.449064    4267 cpu_manager.go:414] "containerMap: removing container" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main"
Oct 30 19:19:24.449125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.449080    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main"
Oct 30 19:19:24.449125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.449113    4267 cpu_manager.go:414] "containerMap: removing container" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main"
Oct 30 19:19:24.449125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.449121    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main"
Oct 30 19:19:24.449225 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.449187    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" containerName="main"
Oct 30 19:19:24.449225 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.449202    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerName="main"
Oct 30 19:19:24.449225 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.449209    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="64927929-46af-4325-80de-45c8eca88e5d" containerName="istio-proxy"
Oct 30 19:19:24.453539 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.453481    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.466125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.466093    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.466125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.466122    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.466338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.466137    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5c5k\" (UniqueName: \"kubernetes.io/projected/78bb5520-dfd6-49c9-8506-0922401452bd-kube-api-access-n5c5k\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.466338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.466151    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w95f\" (UniqueName: \"kubernetes.io/projected/02b3f14f-34ea-4635-9e93-d500debd10d9-kube-api-access-2w95f\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.466338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.466164    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b3f14f-34ea-4635-9e93-d500debd10d9-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.466338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.466177    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.466338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.466188    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78bb5520-dfd6-49c9-8506-0922401452bd-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.466338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.466200    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78bb5520-dfd6-49c9-8506-0922401452bd-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.466338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.466212    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b3f14f-34ea-4635-9e93-d500debd10d9-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:24.506172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.506138    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:19:24.506662 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.506287    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:19:24.539268 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.535008    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"]
Oct 30 19:19:24.545322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.545288    4267 generic.go:356] "Generic (PLEG): container finished" podID="02b3f14f-34ea-4635-9e93-d500debd10d9" containerID="4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810" exitCode=0
Oct 30 19:19:24.545541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.545391    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" event={"ID":"02b3f14f-34ea-4635-9e93-d500debd10d9","Type":"ContainerDied","Data":"4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810"}
Oct 30 19:19:24.545541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.545430    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q" event={"ID":"02b3f14f-34ea-4635-9e93-d500debd10d9","Type":"ContainerDied","Data":"34102637eb0eb5a6044a347e39ed25356deec80d4d8a53f66bb93f09f2af262b"}
Oct 30 19:19:24.545541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.545453    4267 scope.go:122] "RemoveContainer" containerID="4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810"
Oct 30 19:19:24.545740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.545640    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"
Oct 30 19:19:24.552732 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.551606    4267 generic.go:356] "Generic (PLEG): container finished" podID="78bb5520-dfd6-49c9-8506-0922401452bd" containerID="4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f" exitCode=0
Oct 30 19:19:24.552732 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.551675    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"
Oct 30 19:19:24.552732 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.551722    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" event={"ID":"78bb5520-dfd6-49c9-8506-0922401452bd","Type":"ContainerDied","Data":"4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f"}
Oct 30 19:19:24.552732 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.551756    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7" event={"ID":"78bb5520-dfd6-49c9-8506-0922401452bd","Type":"ContainerDied","Data":"6c5d5f46a148bf0df4f653f2ce31a9601230f615f4de0f7c387a20efe5dd7690"}
Oct 30 19:19:24.569385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.569056    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.569385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.569122    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.569385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.569170    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpx96\" (UniqueName: \"kubernetes.io/projected/0467c183-172b-4b8a-a26d-f07d4f4c28f5-kube-api-access-dpx96\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.570781 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.570744    4267 scope.go:122] "RemoveContainer" containerID="4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810"
Oct 30 19:19:24.572317 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:19:24.572167    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810\": container with ID starting with 4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810 not found: ID does not exist" containerID="4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810"
Oct 30 19:19:24.572317 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.572197    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810"} err="failed to get container status \"4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810\": rpc error: code = NotFound desc = could not find container \"4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810\": container with ID starting with 4b2a0dcd9e8314c8d1387b51b752523cea854859e760bd7ee27eb37a2e440810 not found: ID does not exist"
Oct 30 19:19:24.572317 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.572229    4267 scope.go:122] "RemoveContainer" containerID="4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f"
Oct 30 19:19:24.583443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.583409    4267 scope.go:122] "RemoveContainer" containerID="4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f"
Oct 30 19:19:24.583753 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:19:24.583734    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f\": container with ID starting with 4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f not found: ID does not exist" containerID="4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f"
Oct 30 19:19:24.583820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.583764    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f"} err="failed to get container status \"4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f\": rpc error: code = NotFound desc = could not find container \"4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f\": container with ID starting with 4b6dbc5453c7c2591ce2d19052cdd657fc875fcbc9a72a58c9b5e182612d136f not found: ID does not exist"
Oct 30 19:19:24.597286 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.597264    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:24.597418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.597404    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:19:24.670039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.669953    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.670039 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.670010    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpx96\" (UniqueName: \"kubernetes.io/projected/0467c183-172b-4b8a-a26d-f07d4f4c28f5-kube-api-access-dpx96\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.670298 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.670139    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.670804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.670783    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.672677 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.672654    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.693092 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.693069    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"]
Oct 30 19:19:24.696221 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.696198    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-611053/distinct-w5-611053-kserve-cff86dc44-flms7"]
Oct 30 19:19:24.738810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.738775    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpx96\" (UniqueName: \"kubernetes.io/projected/0467c183-172b-4b8a-a26d-f07d4f4c28f5-kube-api-access-dpx96\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.770799 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.770771    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:24.804842 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.804807    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"]
Oct 30 19:19:24.807675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.807641    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-611053/premium-w5-611053-kserve-849cbd585-hnb5q"]
Oct 30 19:19:24.835531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.830408    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8cp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:19:24.835531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:24.830485    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podUID="b4b5cee2-4ba7-4d14-bde2-dd266f4706f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:19:25.179311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:25.179281    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"]
Oct 30 19:19:25.179636 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:19:25.179609    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467c183_172b_4b8a_a26d_f07d4f4c28f5.slice/crio-ab6722e83bc4f5cf7e951c332967dac15b82b4afd4bb879252df2466bb645739 WatchSource:0}: Error finding container ab6722e83bc4f5cf7e951c332967dac15b82b4afd4bb879252df2466bb645739: Status 404 returned error can't find the container with id ab6722e83bc4f5cf7e951c332967dac15b82b4afd4bb879252df2466bb645739
Oct 30 19:19:25.231523 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:25.229622    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b3f14f-34ea-4635-9e93-d500debd10d9" path="/var/lib/kubelet/pods/02b3f14f-34ea-4635-9e93-d500debd10d9/volumes"
Oct 30 19:19:25.231523 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:25.230252    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78bb5520-dfd6-49c9-8506-0922401452bd" path="/var/lib/kubelet/pods/78bb5520-dfd6-49c9-8506-0922401452bd/volumes"
Oct 30 19:19:25.562059 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:25.562012    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7" event={"ID":"0467c183-172b-4b8a-a26d-f07d4f4c28f5","Type":"ContainerStarted","Data":"e0eb9abb6533d1ca4590063683b09c24d14ffa7c1801c8de6f0cd566207a8b90"}
Oct 30 19:19:25.562456 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:25.562078    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7" event={"ID":"0467c183-172b-4b8a-a26d-f07d4f4c28f5","Type":"ContainerStarted","Data":"ab6722e83bc4f5cf7e951c332967dac15b82b4afd4bb879252df2466bb645739"}
Oct 30 19:19:25.694509 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:25.694452    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7" podStartSLOduration=1.694438383 podStartE2EDuration="1.694438383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:25.691340265 +0000 UTC m=+2456.579257354" watchObservedRunningTime="2026-10-30 19:19:25.694438383 +0000 UTC m=+2456.582355471"
Oct 30 19:19:25.827216 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:25.826696    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8cp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:19:25.827216 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:25.826759    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podUID="b4b5cee2-4ba7-4d14-bde2-dd266f4706f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:19:25.846264 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:25.846228    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj" podUID="96769b37-30bd-4d6c-8035-39bffcbc1f34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:26.570552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:26.570466    4267 generic.go:356] "Generic (PLEG): container finished" podID="0467c183-172b-4b8a-a26d-f07d4f4c28f5" containerID="e0eb9abb6533d1ca4590063683b09c24d14ffa7c1801c8de6f0cd566207a8b90" exitCode=0
Oct 30 19:19:26.570925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:26.570563    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7" event={"ID":"0467c183-172b-4b8a-a26d-f07d4f4c28f5","Type":"ContainerDied","Data":"e0eb9abb6533d1ca4590063683b09c24d14ffa7c1801c8de6f0cd566207a8b90"}
Oct 30 19:19:26.747449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:26.747411    4267 patch_prober.go:29] interesting pod/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:19:26.747645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:26.747577    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:19:26.830577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:26.829943    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8cp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:19:26.830577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:26.830012    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podUID="b4b5cee2-4ba7-4d14-bde2-dd266f4706f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:19:27.583118 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.583083    4267 generic.go:356] "Generic (PLEG): container finished" podID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerID="aca9ce4380bab96d9ef5d070ef64c55a1aaa0d6759f91c9ff31e77b7400d857c" exitCode=0
Oct 30 19:19:27.583639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.583160    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" event={"ID":"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2","Type":"ContainerDied","Data":"aca9ce4380bab96d9ef5d070ef64c55a1aaa0d6759f91c9ff31e77b7400d857c"}
Oct 30 19:19:27.583639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.583231    4267 scope.go:122] "RemoveContainer" containerID="8bac6f81dbf151ced9883fa5f0645228bf991f6246d50725d877593efe1d0f27"
Oct 30 19:19:27.666708 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.666672    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 19:19:27.763286 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.763260    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:27.806519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.806464    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-data\") pod \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") "
Oct 30 19:19:27.806678 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.806563    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-podinfo\") pod \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") "
Oct 30 19:19:27.806678 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.806604    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-token\" (UniqueName: \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-token\") pod \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") "
Oct 30 19:19:27.806817 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.806690    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-envoy\") pod \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") "
Oct 30 19:19:27.806817 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.806754    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-socket\") pod \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") "
Oct 30 19:19:27.806817 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.806797    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-certs\") pod \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") "
Oct 30 19:19:27.806980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.806832    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-kube-api-access-8hhtr\" (UniqueName: \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-kube-api-access-8hhtr\") pod \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") "
Oct 30 19:19:27.806980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.806796    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-data" pod "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" (UID: "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:27.806980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.806913    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istiod-ca-cert\") pod \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") "
Oct 30 19:19:27.807169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.806999    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-socket" pod "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" (UID: "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:27.807169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.807037    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-credential-socket\") pod \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\" (UID: \"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2\") "
Oct 30 19:19:27.807169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.807072    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-certs" pod "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" (UID: "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:27.807393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.807280    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istiod-ca-cert" pod "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" (UID: "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:27.807467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.807348    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-credential-socket" pod "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" (UID: "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:27.807559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.807536    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.807637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.807560    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.807637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.807574    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.807637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.807589    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.807637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.807602    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.827533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.826656    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-envoy" pod "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" (UID: "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:27.827533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.827111    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8cp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:19:27.827533 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.827168    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podUID="b4b5cee2-4ba7-4d14-bde2-dd266f4706f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:19:27.834727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.834647    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-podinfo" pod "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" (UID: "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:19:27.834727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.834662    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-token" pod "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" (UID: "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:27.834938 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.834886    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-kube-api-access-8hhtr" pod "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" (UID: "cf09e949-f7ee-4e52-8a6d-a789cf00b7d2"). InnerVolumeSpecName "kube-api-access-8hhtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:27.908277 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.908175    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-openshift-ca\") pod \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") "
Oct 30 19:19:27.908466 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.908314    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-xks-ca\") pod \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") "
Oct 30 19:19:27.908466 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.908347    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0467c183-172b-4b8a-a26d-f07d4f4c28f5-kube-api-access-dpx96\" (UniqueName: \"kubernetes.io/projected/0467c183-172b-4b8a-a26d-f07d4f4c28f5-kube-api-access-dpx96\") pod \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\" (UID: \"0467c183-172b-4b8a-a26d-f07d4f4c28f5\") "
Oct 30 19:19:27.932520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.908632    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-openshift-ca" pod "0467c183-172b-4b8a-a26d-f07d4f4c28f5" (UID: "0467c183-172b-4b8a-a26d-f07d4f4c28f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:27.932520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.908688    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.932520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.908711    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hhtr\" (UniqueName: \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-kube-api-access-8hhtr\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.932520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.908726    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.932520 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.908738    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:27.934945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.934900    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0467c183-172b-4b8a-a26d-f07d4f4c28f5-kube-api-access-dpx96" pod "0467c183-172b-4b8a-a26d-f07d4f4c28f5" (UID: "0467c183-172b-4b8a-a26d-f07d4f4c28f5"). InnerVolumeSpecName "kube-api-access-dpx96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:27.936873 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:27.936839    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-xks-ca" pod "0467c183-172b-4b8a-a26d-f07d4f4c28f5" (UID: "0467c183-172b-4b8a-a26d-f07d4f4c28f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:28.034519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.033319    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:28.034519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.033370    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpx96\" (UniqueName: \"kubernetes.io/projected/0467c183-172b-4b8a-a26d-f07d4f4c28f5-kube-api-access-dpx96\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:28.034519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.033387    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0467c183-172b-4b8a-a26d-f07d4f4c28f5-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:28.591736 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.591708    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"
Oct 30 19:19:28.592223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.591733    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49" event={"ID":"cf09e949-f7ee-4e52-8a6d-a789cf00b7d2","Type":"ContainerDied","Data":"c5e9a61ab3f962f3c2d76710da5bc491d3a1eff2e05477eae1aec03cbf61e6f8"}
Oct 30 19:19:28.592223 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.591804    4267 scope.go:122] "RemoveContainer" containerID="aca9ce4380bab96d9ef5d070ef64c55a1aaa0d6759f91c9ff31e77b7400d857c"
Oct 30 19:19:28.593365 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.593343    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"
Oct 30 19:19:28.593482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.593364    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7" event={"ID":"0467c183-172b-4b8a-a26d-f07d4f4c28f5","Type":"ContainerDied","Data":"ab6722e83bc4f5cf7e951c332967dac15b82b4afd4bb879252df2466bb645739"}
Oct 30 19:19:28.593482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.593411    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab6722e83bc4f5cf7e951c332967dac15b82b4afd4bb879252df2466bb645739"
Oct 30 19:19:28.828252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.828201    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"]
Oct 30 19:19:28.835517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.830910    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8cp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:19:28.835517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.830972    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podUID="b4b5cee2-4ba7-4d14-bde2-dd266f4706f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:19:28.861174 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:28.861134    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1290a6-openshift-default-7f5f577cf-g2p49"]
Oct 30 19:19:29.220862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.220825    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:19:29.221999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.221976    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" path="/var/lib/kubelet/pods/cf09e949-f7ee-4e52-8a6d-a789cf00b7d2/volumes"
Oct 30 19:19:29.348409 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.348318    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-kube-api-access-m2kkh\" (UniqueName: \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-kube-api-access-m2kkh\") pod \"5f6af68d-aabc-4ced-b194-395f48c4e36c\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") "
Oct 30 19:19:29.348598 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.348408    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-token\" (UniqueName: \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-token\") pod \"5f6af68d-aabc-4ced-b194-395f48c4e36c\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") "
Oct 30 19:19:29.348598 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.348526    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-socket\") pod \"5f6af68d-aabc-4ced-b194-395f48c4e36c\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") "
Oct 30 19:19:29.348598 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.348562    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-envoy\") pod \"5f6af68d-aabc-4ced-b194-395f48c4e36c\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") "
Oct 30 19:19:29.348598 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.348594    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6af68d-aabc-4ced-b194-395f48c4e36c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6af68d-aabc-4ced-b194-395f48c4e36c-istiod-ca-cert\") pod \"5f6af68d-aabc-4ced-b194-395f48c4e36c\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") "
Oct 30 19:19:29.348831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.348642    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-data\") pod \"5f6af68d-aabc-4ced-b194-395f48c4e36c\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") "
Oct 30 19:19:29.348831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.348670    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-certs\") pod \"5f6af68d-aabc-4ced-b194-395f48c4e36c\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") "
Oct 30 19:19:29.348831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.348696    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-podinfo\") pod \"5f6af68d-aabc-4ced-b194-395f48c4e36c\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") "
Oct 30 19:19:29.348831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.348796    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-socket" pod "5f6af68d-aabc-4ced-b194-395f48c4e36c" (UID: "5f6af68d-aabc-4ced-b194-395f48c4e36c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:29.348831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.348815    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-credential-socket\") pod \"5f6af68d-aabc-4ced-b194-395f48c4e36c\" (UID: \"5f6af68d-aabc-4ced-b194-395f48c4e36c\") "
Oct 30 19:19:29.349107 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.349005    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-data" pod "5f6af68d-aabc-4ced-b194-395f48c4e36c" (UID: "5f6af68d-aabc-4ced-b194-395f48c4e36c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:29.349158 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.349138    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-certs" pod "5f6af68d-aabc-4ced-b194-395f48c4e36c" (UID: "5f6af68d-aabc-4ced-b194-395f48c4e36c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:29.349284 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.349261    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-credential-socket" pod "5f6af68d-aabc-4ced-b194-395f48c4e36c" (UID: "5f6af68d-aabc-4ced-b194-395f48c4e36c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:29.349420 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.349400    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:29.349482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.349427    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:29.349482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.349444    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:29.349482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.349460    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:29.349801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.349755    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6af68d-aabc-4ced-b194-395f48c4e36c-istiod-ca-cert" pod "5f6af68d-aabc-4ced-b194-395f48c4e36c" (UID: "5f6af68d-aabc-4ced-b194-395f48c4e36c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:29.351943 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.351529    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-envoy" pod "5f6af68d-aabc-4ced-b194-395f48c4e36c" (UID: "5f6af68d-aabc-4ced-b194-395f48c4e36c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:29.351943 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.351784    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-token" pod "5f6af68d-aabc-4ced-b194-395f48c4e36c" (UID: "5f6af68d-aabc-4ced-b194-395f48c4e36c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:29.351943 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.351810    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-kube-api-access-m2kkh" pod "5f6af68d-aabc-4ced-b194-395f48c4e36c" (UID: "5f6af68d-aabc-4ced-b194-395f48c4e36c"). InnerVolumeSpecName "kube-api-access-m2kkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:29.352358 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.352325    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-podinfo" pod "5f6af68d-aabc-4ced-b194-395f48c4e36c" (UID: "5f6af68d-aabc-4ced-b194-395f48c4e36c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:19:29.450725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.450682    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:29.450725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.450727    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f6af68d-aabc-4ced-b194-395f48c4e36c-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:29.450947 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.450744    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:29.450947 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.450760    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2kkh\" (UniqueName: \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-kube-api-access-m2kkh\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:29.450947 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.450775    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f6af68d-aabc-4ced-b194-395f48c4e36c-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:29.601857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.601768    4267 generic.go:356] "Generic (PLEG): container finished" podID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerID="93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f" exitCode=0
Oct 30 19:19:29.601857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.601822    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"
Oct 30 19:19:29.602386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.601861    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" event={"ID":"5f6af68d-aabc-4ced-b194-395f48c4e36c","Type":"ContainerDied","Data":"93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f"}
Oct 30 19:19:29.602386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.601929    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25" event={"ID":"5f6af68d-aabc-4ced-b194-395f48c4e36c","Type":"ContainerDied","Data":"43d898e7193919a5562798750e89550795513b91238b60cb9e6120031523e9ce"}
Oct 30 19:19:29.602386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.601964    4267 scope.go:122] "RemoveContainer" containerID="93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f"
Oct 30 19:19:29.616058 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.616029    4267 scope.go:122] "RemoveContainer" containerID="fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f"
Oct 30 19:19:29.632742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.632709    4267 scope.go:122] "RemoveContainer" containerID="93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f"
Oct 30 19:19:29.633168 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:19:29.633141    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f\": container with ID starting with 93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f not found: ID does not exist" containerID="93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f"
Oct 30 19:19:29.633300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.633173    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f"} err="failed to get container status \"93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f\": rpc error: code = NotFound desc = could not find container \"93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f\": container with ID starting with 93c2f85b5d36c55ca92c7e1c3c2d59c50c9ced4892195e3c104649f3b53a250f not found: ID does not exist"
Oct 30 19:19:29.633300 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.633195    4267 scope.go:122] "RemoveContainer" containerID="fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f"
Oct 30 19:19:29.633579 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:19:29.633553    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f\": container with ID starting with fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f not found: ID does not exist" containerID="fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f"
Oct 30 19:19:29.633657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.633593    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f"} err="failed to get container status \"fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f\": rpc error: code = NotFound desc = could not find container \"fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f\": container with ID starting with fa7e18f3f67618827774459300a71cd63dcabe86c3ec29eeca1f257a4128793f not found: ID does not exist"
Oct 30 19:19:29.770743 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.770710    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"]
Oct 30 19:19:29.820068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.820033    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36d676-openshift-default-6dc96c545c-ffr25"]
Oct 30 19:19:29.826338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.826304    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8cp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:19:29.826508 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:29.826367    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podUID="b4b5cee2-4ba7-4d14-bde2-dd266f4706f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:19:30.826410 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:30.826370    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8cp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 19:19:30.826900 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:30.826455    4267 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podUID="b4b5cee2-4ba7-4d14-bde2-dd266f4706f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 19:19:31.225143 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:31.225097    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" path="/var/lib/kubelet/pods/5f6af68d-aabc-4ced-b194-395f48c4e36c/volumes"
Oct 30 19:19:31.455470 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:31.455424    4267 scope.go:122] "RemoveContainer" containerID="42a3e5c7393fa18d53f15c5419f0aae9d32365581af5c5038889e3ac15e47697"
Oct 30 19:19:31.468361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:31.468330    4267 scope.go:122] "RemoveContainer" containerID="032a352a2235eb29106c69392901a1550e27078c18ca8261f358462ec9924770"
Oct 30 19:19:31.830880 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:31.830806    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:31.831323 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:31.831163    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:31.832049 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:31.832029    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28"
Oct 30 19:19:35.841747 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:35.841697    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj" podUID="96769b37-30bd-4d6c-8035-39bffcbc1f34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:19:36.947153 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:36.947114    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"]
Oct 30 19:19:40.688548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:40.688474    4267 generic.go:356] "Generic (PLEG): container finished" podID="96769b37-30bd-4d6c-8035-39bffcbc1f34" containerID="cde569a9a9062ebf30f597444247435bef27b136d9c148ed33b13c5816ceb5d7" exitCode=0
Oct 30 19:19:40.689041 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:40.688648    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj" event={"ID":"96769b37-30bd-4d6c-8035-39bffcbc1f34","Type":"ContainerDied","Data":"cde569a9a9062ebf30f597444247435bef27b136d9c148ed33b13c5816ceb5d7"}
Oct 30 19:19:40.790597 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:40.789854    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 19:19:40.886964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:40.886929    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96769b37-30bd-4d6c-8035-39bffcbc1f34-kube-api-access-cztv7\" (UniqueName: \"kubernetes.io/projected/96769b37-30bd-4d6c-8035-39bffcbc1f34-kube-api-access-cztv7\") pod \"96769b37-30bd-4d6c-8035-39bffcbc1f34\" (UID: \"96769b37-30bd-4d6c-8035-39bffcbc1f34\") "
Oct 30 19:19:40.887169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:40.887091    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96769b37-30bd-4d6c-8035-39bffcbc1f34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96769b37-30bd-4d6c-8035-39bffcbc1f34-plugins-config-volume\") pod \"96769b37-30bd-4d6c-8035-39bffcbc1f34\" (UID: \"96769b37-30bd-4d6c-8035-39bffcbc1f34\") "
Oct 30 19:19:40.887476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:40.887441    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96769b37-30bd-4d6c-8035-39bffcbc1f34-plugins-config-volume" pod "96769b37-30bd-4d6c-8035-39bffcbc1f34" (UID: "96769b37-30bd-4d6c-8035-39bffcbc1f34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:40.889181 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:40.889160    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96769b37-30bd-4d6c-8035-39bffcbc1f34-kube-api-access-cztv7" pod "96769b37-30bd-4d6c-8035-39bffcbc1f34" (UID: "96769b37-30bd-4d6c-8035-39bffcbc1f34"). InnerVolumeSpecName "kube-api-access-cztv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:40.989041 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:40.988956    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96769b37-30bd-4d6c-8035-39bffcbc1f34-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:40.989041 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:40.988991    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cztv7\" (UniqueName: \"kubernetes.io/projected/96769b37-30bd-4d6c-8035-39bffcbc1f34-kube-api-access-cztv7\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:41.695485 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:41.695456    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"
Oct 30 19:19:41.695939 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:41.695508    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj" event={"ID":"96769b37-30bd-4d6c-8035-39bffcbc1f34","Type":"ContainerDied","Data":"67c4d5d1e68858f4bbbbd47621abfe264371e3db61f3ab24b9180efd2b2a5780"}
Oct 30 19:19:41.695939 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:41.695578    4267 scope.go:122] "RemoveContainer" containerID="cde569a9a9062ebf30f597444247435bef27b136d9c148ed33b13c5816ceb5d7"
Oct 30 19:19:41.881175 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:41.881129    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"]
Oct 30 19:19:41.886174 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:41.886125    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-611053-569b766769-qjkgj"]
Oct 30 19:19:41.947569 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:41.947448    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s" podUID="f2c2c888-6a51-447e-aaa9-33a834dcf653" containerName="payload-pre-processing" containerID="cri-o://7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef" gracePeriod=25
Oct 30 19:19:42.213240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.213131    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:19:42.302612 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.302577    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c2c888-6a51-447e-aaa9-33a834dcf653-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c2c888-6a51-447e-aaa9-33a834dcf653-plugins-config-volume\") pod \"f2c2c888-6a51-447e-aaa9-33a834dcf653\" (UID: \"f2c2c888-6a51-447e-aaa9-33a834dcf653\") "
Oct 30 19:19:42.302830 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.302734    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c2c888-6a51-447e-aaa9-33a834dcf653-kube-api-access-vk9jd\" (UniqueName: \"kubernetes.io/projected/f2c2c888-6a51-447e-aaa9-33a834dcf653-kube-api-access-vk9jd\") pod \"f2c2c888-6a51-447e-aaa9-33a834dcf653\" (UID: \"f2c2c888-6a51-447e-aaa9-33a834dcf653\") "
Oct 30 19:19:42.303012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.302984    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c2c888-6a51-447e-aaa9-33a834dcf653-plugins-config-volume" pod "f2c2c888-6a51-447e-aaa9-33a834dcf653" (UID: "f2c2c888-6a51-447e-aaa9-33a834dcf653"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:42.303127 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.303107    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c2c888-6a51-447e-aaa9-33a834dcf653-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:42.304909 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.304887    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c2c888-6a51-447e-aaa9-33a834dcf653-kube-api-access-vk9jd" pod "f2c2c888-6a51-447e-aaa9-33a834dcf653" (UID: "f2c2c888-6a51-447e-aaa9-33a834dcf653"). InnerVolumeSpecName "kube-api-access-vk9jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:42.404641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.404602    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk9jd\" (UniqueName: \"kubernetes.io/projected/f2c2c888-6a51-447e-aaa9-33a834dcf653-kube-api-access-vk9jd\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:42.704136 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.704094    4267 generic.go:356] "Generic (PLEG): container finished" podID="f2c2c888-6a51-447e-aaa9-33a834dcf653" containerID="7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef" exitCode=0
Oct 30 19:19:42.704714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.704146    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"
Oct 30 19:19:42.704714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.704157    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s" event={"ID":"f2c2c888-6a51-447e-aaa9-33a834dcf653","Type":"ContainerDied","Data":"7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef"}
Oct 30 19:19:42.704714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.704203    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s" event={"ID":"f2c2c888-6a51-447e-aaa9-33a834dcf653","Type":"ContainerDied","Data":"e8f30bf0135025cf34eff60652e1714bab7633c2aa72d49810b846c43e2dac66"}
Oct 30 19:19:42.704714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.704227    4267 scope.go:122] "RemoveContainer" containerID="7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef"
Oct 30 19:19:42.714656 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.714586    4267 scope.go:122] "RemoveContainer" containerID="7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef"
Oct 30 19:19:42.714926 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:19:42.714907    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef\": container with ID starting with 7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef not found: ID does not exist" containerID="7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef"
Oct 30 19:19:42.714984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.714939    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef"} err="failed to get container status \"7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef\": rpc error: code = NotFound desc = could not find container \"7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef\": container with ID starting with 7234571074f061c02fa1fe25a5544851898c46cf8431412d063b6dc9ac39a0ef not found: ID does not exist"
Oct 30 19:19:42.869006 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.868966    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"]
Oct 30 19:19:42.918687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:42.918647    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-732a3f-57b9b89895-xns4s"]
Oct 30 19:19:43.219943 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:43.219908    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96769b37-30bd-4d6c-8035-39bffcbc1f34" path="/var/lib/kubelet/pods/96769b37-30bd-4d6c-8035-39bffcbc1f34/volumes"
Oct 30 19:19:43.220369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:43.220355    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c2c888-6a51-447e-aaa9-33a834dcf653" path="/var/lib/kubelet/pods/f2c2c888-6a51-447e-aaa9-33a834dcf653/volumes"
Oct 30 19:19:46.091156 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091118    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"]
Oct 30 19:19:46.091580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091543    4267 cpu_manager.go:414] "containerMap: removing container" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy"
Oct 30 19:19:46.091580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091556    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy"
Oct 30 19:19:46.091580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091570    4267 cpu_manager.go:414] "containerMap: removing container" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy"
Oct 30 19:19:46.091580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091575    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091587    4267 cpu_manager.go:414] "containerMap: removing container" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091593    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091605    4267 cpu_manager.go:414] "containerMap: removing container" podUID="0467c183-172b-4b8a-a26d-f07d4f4c28f5" containerName="revoke-keys"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091610    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0467c183-172b-4b8a-a26d-f07d4f4c28f5" containerName="revoke-keys"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091621    4267 cpu_manager.go:414] "containerMap: removing container" podUID="f2c2c888-6a51-447e-aaa9-33a834dcf653" containerName="payload-pre-processing"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091626    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c2c888-6a51-447e-aaa9-33a834dcf653" containerName="payload-pre-processing"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091637    4267 cpu_manager.go:414] "containerMap: removing container" podUID="96769b37-30bd-4d6c-8035-39bffcbc1f34" containerName="payload-processing"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091642    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96769b37-30bd-4d6c-8035-39bffcbc1f34" containerName="payload-processing"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091702    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c2c888-6a51-447e-aaa9-33a834dcf653" containerName="payload-pre-processing"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091711    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="96769b37-30bd-4d6c-8035-39bffcbc1f34" containerName="payload-processing"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091718    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091724    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091732    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy"
Oct 30 19:19:46.091738 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.091740    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="0467c183-172b-4b8a-a26d-f07d4f4c28f5" containerName="revoke-keys"
Oct 30 19:19:46.096216 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.096199    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.144761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.144728    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rglt\" (UniqueName: \"kubernetes.io/projected/b9a51ae4-9150-4918-bc06-2876614af3a1-kube-api-access-7rglt\") pod \"maas-api-revoke-sub-32f8257b-wxf8s\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.144761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.144765    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32f8257b-wxf8s\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.144953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.144915    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32f8257b-wxf8s\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.150856 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.150817    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:19:46.153975 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.153949    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"]
Oct 30 19:19:46.154521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.154479    4267 cpu_manager.go:414] "containerMap: removing container" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy"
Oct 30 19:19:46.154521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.154517    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6af68d-aabc-4ced-b194-395f48c4e36c" containerName="istio-proxy"
Oct 30 19:19:46.154690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.154650    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf09e949-f7ee-4e52-8a6d-a789cf00b7d2" containerName="istio-proxy"
Oct 30 19:19:46.157865 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.157841    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.200625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.200591    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:46.200789 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.200635    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:19:46.200789 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.200658    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:19:46.204890 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.204870    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"]
Oct 30 19:19:46.218119 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.218092    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"]
Oct 30 19:19:46.245554 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.245516    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rglt\" (UniqueName: \"kubernetes.io/projected/b9a51ae4-9150-4918-bc06-2876614af3a1-kube-api-access-7rglt\") pod \"maas-api-revoke-sub-32f8257b-wxf8s\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.245759 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.245639    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32f8257b-wxf8s\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.245836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.245794    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32f8257b-wxf8s\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.245836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.245830    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a0ae7d-4ppnc\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.245940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.245876    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj7jh\" (UniqueName: \"kubernetes.io/projected/62c1a2ea-9388-4576-8982-4eabbce9c52e-kube-api-access-fj7jh\") pod \"maas-api-revoke-sub-81a0ae7d-4ppnc\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.246002 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.245966    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a0ae7d-4ppnc\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.246559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.246528    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32f8257b-wxf8s\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.248392 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.248360    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32f8257b-wxf8s\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.306568 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.306535    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rglt\" (UniqueName: \"kubernetes.io/projected/b9a51ae4-9150-4918-bc06-2876614af3a1-kube-api-access-7rglt\") pod \"maas-api-revoke-sub-32f8257b-wxf8s\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.347159 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.347073    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a0ae7d-4ppnc\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.347159 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.347142    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj7jh\" (UniqueName: \"kubernetes.io/projected/62c1a2ea-9388-4576-8982-4eabbce9c52e-kube-api-access-fj7jh\") pod \"maas-api-revoke-sub-81a0ae7d-4ppnc\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.347367 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.347178    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a0ae7d-4ppnc\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.347937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.347911    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a0ae7d-4ppnc\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.349464 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.349443    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a0ae7d-4ppnc\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.404368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.404333    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj7jh\" (UniqueName: \"kubernetes.io/projected/62c1a2ea-9388-4576-8982-4eabbce9c52e-kube-api-access-fj7jh\") pod \"maas-api-revoke-sub-81a0ae7d-4ppnc\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.409993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.409963    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:46.477815 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.475662    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:46.785086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.785045    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"]
Oct 30 19:19:46.788768 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:19:46.788741    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a51ae4_9150_4918_bc06_2876614af3a1.slice/crio-8ea0067ed6514e2758d87a810043079cb388772f118295b7ac3c89f92437e523 WatchSource:0}: Error finding container 8ea0067ed6514e2758d87a810043079cb388772f118295b7ac3c89f92437e523: Status 404 returned error can't find the container with id 8ea0067ed6514e2758d87a810043079cb388772f118295b7ac3c89f92437e523
Oct 30 19:19:46.887353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:46.887327    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"]
Oct 30 19:19:46.888302 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:19:46.888275    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c1a2ea_9388_4576_8982_4eabbce9c52e.slice/crio-8ee1120812b8fba281954760c0e6bb9c93fd590dbeb79458d12cd8bd80e3cc7a WatchSource:0}: Error finding container 8ee1120812b8fba281954760c0e6bb9c93fd590dbeb79458d12cd8bd80e3cc7a: Status 404 returned error can't find the container with id 8ee1120812b8fba281954760c0e6bb9c93fd590dbeb79458d12cd8bd80e3cc7a
Oct 30 19:19:47.753458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:47.753421    4267 generic.go:356] "Generic (PLEG): container finished" podID="b9a51ae4-9150-4918-bc06-2876614af3a1" containerID="4f46c69102443f81c62b949e523993f2bacbe392612c0b689062b633f68d6a94" exitCode=0
Oct 30 19:19:47.753963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:47.753535    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s" event={"ID":"b9a51ae4-9150-4918-bc06-2876614af3a1","Type":"ContainerDied","Data":"4f46c69102443f81c62b949e523993f2bacbe392612c0b689062b633f68d6a94"}
Oct 30 19:19:47.753963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:47.753605    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s" event={"ID":"b9a51ae4-9150-4918-bc06-2876614af3a1","Type":"ContainerStarted","Data":"8ea0067ed6514e2758d87a810043079cb388772f118295b7ac3c89f92437e523"}
Oct 30 19:19:47.755255 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:47.755231    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc" event={"ID":"62c1a2ea-9388-4576-8982-4eabbce9c52e","Type":"ContainerStarted","Data":"c540b4c67efe7aa334f7a2cb2666069a8d6085c257afaaab258bd7f49fba070e"}
Oct 30 19:19:47.755384 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:47.755273    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc" event={"ID":"62c1a2ea-9388-4576-8982-4eabbce9c52e","Type":"ContainerStarted","Data":"8ee1120812b8fba281954760c0e6bb9c93fd590dbeb79458d12cd8bd80e3cc7a"}
Oct 30 19:19:47.987558 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:47.987487    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc" podStartSLOduration=1.9874703070000002 podStartE2EDuration="1.987470307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:47.984913419 +0000 UTC m=+2478.872830507" watchObservedRunningTime="2026-10-30 19:19:47.987470307 +0000 UTC m=+2478.875387396"
Oct 30 19:19:48.763326 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:48.763290    4267 generic.go:356] "Generic (PLEG): container finished" podID="62c1a2ea-9388-4576-8982-4eabbce9c52e" containerID="c540b4c67efe7aa334f7a2cb2666069a8d6085c257afaaab258bd7f49fba070e" exitCode=0
Oct 30 19:19:48.763835 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:48.763378    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc" event={"ID":"62c1a2ea-9388-4576-8982-4eabbce9c52e","Type":"ContainerDied","Data":"c540b4c67efe7aa334f7a2cb2666069a8d6085c257afaaab258bd7f49fba070e"}
Oct 30 19:19:48.911961 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:48.911927    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:48.977445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:48.977417    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a51ae4-9150-4918-bc06-2876614af3a1-kube-api-access-7rglt\" (UniqueName: \"kubernetes.io/projected/b9a51ae4-9150-4918-bc06-2876614af3a1-kube-api-access-7rglt\") pod \"b9a51ae4-9150-4918-bc06-2876614af3a1\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") "
Oct 30 19:19:48.977653 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:48.977465    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-openshift-ca\") pod \"b9a51ae4-9150-4918-bc06-2876614af3a1\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") "
Oct 30 19:19:48.977653 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:48.977533    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-xks-ca\") pod \"b9a51ae4-9150-4918-bc06-2876614af3a1\" (UID: \"b9a51ae4-9150-4918-bc06-2876614af3a1\") "
Oct 30 19:19:48.977912 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:48.977884    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-openshift-ca" pod "b9a51ae4-9150-4918-bc06-2876614af3a1" (UID: "b9a51ae4-9150-4918-bc06-2876614af3a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:48.979661 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:48.979637    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-xks-ca" pod "b9a51ae4-9150-4918-bc06-2876614af3a1" (UID: "b9a51ae4-9150-4918-bc06-2876614af3a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:48.979764 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:48.979747    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a51ae4-9150-4918-bc06-2876614af3a1-kube-api-access-7rglt" pod "b9a51ae4-9150-4918-bc06-2876614af3a1" (UID: "b9a51ae4-9150-4918-bc06-2876614af3a1"). InnerVolumeSpecName "kube-api-access-7rglt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:49.078512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.078390    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rglt\" (UniqueName: \"kubernetes.io/projected/b9a51ae4-9150-4918-bc06-2876614af3a1-kube-api-access-7rglt\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:49.078512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.078431    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:49.078512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.078446    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9a51ae4-9150-4918-bc06-2876614af3a1-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:49.771288 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.771258    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"
Oct 30 19:19:49.771288 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.771237    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s" event={"ID":"b9a51ae4-9150-4918-bc06-2876614af3a1","Type":"ContainerDied","Data":"8ea0067ed6514e2758d87a810043079cb388772f118295b7ac3c89f92437e523"}
Oct 30 19:19:49.771775 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.771312    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ea0067ed6514e2758d87a810043079cb388772f118295b7ac3c89f92437e523"
Oct 30 19:19:49.915651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.915617    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:49.987303 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.987261    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-xks-ca\") pod \"62c1a2ea-9388-4576-8982-4eabbce9c52e\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") "
Oct 30 19:19:49.987511 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.987341    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c1a2ea-9388-4576-8982-4eabbce9c52e-kube-api-access-fj7jh\" (UniqueName: \"kubernetes.io/projected/62c1a2ea-9388-4576-8982-4eabbce9c52e-kube-api-access-fj7jh\") pod \"62c1a2ea-9388-4576-8982-4eabbce9c52e\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") "
Oct 30 19:19:49.987511 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.987478    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-openshift-ca\") pod \"62c1a2ea-9388-4576-8982-4eabbce9c52e\" (UID: \"62c1a2ea-9388-4576-8982-4eabbce9c52e\") "
Oct 30 19:19:49.987814 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.987791    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-openshift-ca" pod "62c1a2ea-9388-4576-8982-4eabbce9c52e" (UID: "62c1a2ea-9388-4576-8982-4eabbce9c52e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:49.989353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.989324    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-xks-ca" pod "62c1a2ea-9388-4576-8982-4eabbce9c52e" (UID: "62c1a2ea-9388-4576-8982-4eabbce9c52e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:49.989482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:49.989463    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c1a2ea-9388-4576-8982-4eabbce9c52e-kube-api-access-fj7jh" pod "62c1a2ea-9388-4576-8982-4eabbce9c52e" (UID: "62c1a2ea-9388-4576-8982-4eabbce9c52e"). InnerVolumeSpecName "kube-api-access-fj7jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:50.088341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:50.088253    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:50.088341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:50.088287    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj7jh\" (UniqueName: \"kubernetes.io/projected/62c1a2ea-9388-4576-8982-4eabbce9c52e-kube-api-access-fj7jh\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:50.088341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:50.088298    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c1a2ea-9388-4576-8982-4eabbce9c52e-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:19:50.158345 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:19:50.158299    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:19:50.779213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:50.779180    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"
Oct 30 19:19:50.779213 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:50.779204    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc" event={"ID":"62c1a2ea-9388-4576-8982-4eabbce9c52e","Type":"ContainerDied","Data":"8ee1120812b8fba281954760c0e6bb9c93fd590dbeb79458d12cd8bd80e3cc7a"}
Oct 30 19:19:50.779702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:50.779249    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ee1120812b8fba281954760c0e6bb9c93fd590dbeb79458d12cd8bd80e3cc7a"
Oct 30 19:19:53.247483 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:53.247419    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"]
Oct 30 19:19:53.301838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:53.301789    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-732a3f-n89r7"]
Oct 30 19:19:55.220592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:19:55.220549    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0467c183-172b-4b8a-a26d-f07d4f4c28f5" path="/var/lib/kubelet/pods/0467c183-172b-4b8a-a26d-f07d4f4c28f5/volumes"
Oct 30 19:20:00.804955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.804911    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"]
Oct 30 19:20:00.805385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.805309    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b9a51ae4-9150-4918-bc06-2876614af3a1" containerName="revoke-keys"
Oct 30 19:20:00.805385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.805320    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a51ae4-9150-4918-bc06-2876614af3a1" containerName="revoke-keys"
Oct 30 19:20:00.805385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.805334    4267 cpu_manager.go:414] "containerMap: removing container" podUID="62c1a2ea-9388-4576-8982-4eabbce9c52e" containerName="revoke-keys"
Oct 30 19:20:00.805385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.805339    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c1a2ea-9388-4576-8982-4eabbce9c52e" containerName="revoke-keys"
Oct 30 19:20:00.805572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.805412    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a51ae4-9150-4918-bc06-2876614af3a1" containerName="revoke-keys"
Oct 30 19:20:00.805572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.805422    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c1a2ea-9388-4576-8982-4eabbce9c52e" containerName="revoke-keys"
Oct 30 19:20:00.808457 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.808438    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:00.862757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.862707    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:00.866539 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.866511    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"]
Oct 30 19:20:00.910086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.910048    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n7q7\" (UniqueName: \"kubernetes.io/projected/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-kube-api-access-7n7q7\") pod \"maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:00.910261 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.910128    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:00.910308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.910249    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:00.912219 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.912201    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:20:00.912219 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.912220    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:20:00.912363 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:00.912316    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:20:01.011754 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.011696    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:01.011979 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.011777    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:01.012056 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.012006    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n7q7\" (UniqueName: \"kubernetes.io/projected/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-kube-api-access-7n7q7\") pod \"maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:01.012409 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.012375    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:01.014271 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.014251    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:01.069257 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.069189    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n7q7\" (UniqueName: \"kubernetes.io/projected/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-kube-api-access-7n7q7\") pod \"maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:01.121858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.121822    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:01.508315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.508283    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"]
Oct 30 19:20:01.509152 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:20:01.509121    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea552f9_ef32_4d41_9d2d_5429fc3f2049.slice/crio-c70fe206e1c09973beb4e8a669a6170605674647f9f9bef338ce7bc3e3adeaf5 WatchSource:0}: Error finding container c70fe206e1c09973beb4e8a669a6170605674647f9f9bef338ce7bc3e3adeaf5: Status 404 returned error can't find the container with id c70fe206e1c09973beb4e8a669a6170605674647f9f9bef338ce7bc3e3adeaf5
Oct 30 19:20:01.851346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.851295    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4" event={"ID":"3ea552f9-ef32-4d41-9d2d-5429fc3f2049","Type":"ContainerStarted","Data":"1b20d739a74b1fc8c59f8be672e8dbb274d2f76e48f85436cf90095485716bc7"}
Oct 30 19:20:01.851782 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.851390    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4" event={"ID":"3ea552f9-ef32-4d41-9d2d-5429fc3f2049","Type":"ContainerStarted","Data":"c70fe206e1c09973beb4e8a669a6170605674647f9f9bef338ce7bc3e3adeaf5"}
Oct 30 19:20:01.965263 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:01.965196    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4" podStartSLOduration=1.965176167 podStartE2EDuration="1.965176167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:01.964209198 +0000 UTC m=+2492.852126288" watchObservedRunningTime="2026-10-30 19:20:01.965176167 +0000 UTC m=+2492.853093258"
Oct 30 19:20:02.859000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:02.858962    4267 generic.go:356] "Generic (PLEG): container finished" podID="3ea552f9-ef32-4d41-9d2d-5429fc3f2049" containerID="1b20d739a74b1fc8c59f8be672e8dbb274d2f76e48f85436cf90095485716bc7" exitCode=0
Oct 30 19:20:02.859546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:02.859045    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4" event={"ID":"3ea552f9-ef32-4d41-9d2d-5429fc3f2049","Type":"ContainerDied","Data":"1b20d739a74b1fc8c59f8be672e8dbb274d2f76e48f85436cf90095485716bc7"}
Oct 30 19:20:04.006105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.006073    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:04.144073 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.143989    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-xks-ca\") pod \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") "
Oct 30 19:20:04.144245 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.144078    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-kube-api-access-7n7q7\" (UniqueName: \"kubernetes.io/projected/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-kube-api-access-7n7q7\") pod \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") "
Oct 30 19:20:04.144245 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.144211    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-openshift-ca\") pod \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\" (UID: \"3ea552f9-ef32-4d41-9d2d-5429fc3f2049\") "
Oct 30 19:20:04.144603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.144579    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-openshift-ca" pod "3ea552f9-ef32-4d41-9d2d-5429fc3f2049" (UID: "3ea552f9-ef32-4d41-9d2d-5429fc3f2049"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:04.146248 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.146224    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-xks-ca" pod "3ea552f9-ef32-4d41-9d2d-5429fc3f2049" (UID: "3ea552f9-ef32-4d41-9d2d-5429fc3f2049"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:04.146353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.146227    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-kube-api-access-7n7q7" pod "3ea552f9-ef32-4d41-9d2d-5429fc3f2049" (UID: "3ea552f9-ef32-4d41-9d2d-5429fc3f2049"). InnerVolumeSpecName "kube-api-access-7n7q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:04.245460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.245419    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:04.245460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.245453    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n7q7\" (UniqueName: \"kubernetes.io/projected/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-kube-api-access-7n7q7\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:04.245460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.245465    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea552f9-ef32-4d41-9d2d-5429fc3f2049-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:04.872352 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.872308    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4" event={"ID":"3ea552f9-ef32-4d41-9d2d-5429fc3f2049","Type":"ContainerDied","Data":"c70fe206e1c09973beb4e8a669a6170605674647f9f9bef338ce7bc3e3adeaf5"}
Oct 30 19:20:04.872352 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.872357    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c70fe206e1c09973beb4e8a669a6170605674647f9f9bef338ce7bc3e3adeaf5"
Oct 30 19:20:04.872615 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:04.872319    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"
Oct 30 19:20:05.494290 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:05.494252    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"]
Oct 30 19:20:12.340798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:12.340760    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"]
Oct 30 19:20:17.341583 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:17.341531    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m" podUID="53707cb2-f063-4273-bd83-5dc56d5e2c77" containerName="payload-pre-processing" containerID="cri-o://ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190" gracePeriod=25
Oct 30 19:20:17.905662 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:17.905627    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"]
Oct 30 19:20:17.906047 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:17.906034    4267 cpu_manager.go:414] "containerMap: removing container" podUID="3ea552f9-ef32-4d41-9d2d-5429fc3f2049" containerName="revoke-keys"
Oct 30 19:20:17.906099 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:17.906048    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea552f9-ef32-4d41-9d2d-5429fc3f2049" containerName="revoke-keys"
Oct 30 19:20:17.906142 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:17.906132    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea552f9-ef32-4d41-9d2d-5429fc3f2049" containerName="revoke-keys"
Oct 30 19:20:17.909262 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:17.909242    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:17.957807 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:17.957767    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"]
Oct 30 19:20:17.961248 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:17.961233    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.004558 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.004524    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:20:18.007540 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.007487    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81255a89-fdlp5\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.007737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.007713    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5babae40-nqjmg\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:18.007843 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.007741    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5babae40-nqjmg\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:18.007904 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.007847    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm78t\" (UniqueName: \"kubernetes.io/projected/45ffcbb7-526d-4a6c-842f-d7112cd0d942-kube-api-access-gm78t\") pod \"maas-api-revoke-sub-81255a89-fdlp5\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.007944 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.007899    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"]
Oct 30 19:20:18.007987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.007906    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqcx\" (UniqueName: \"kubernetes.io/projected/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-kube-api-access-2vqcx\") pod \"maas-api-revoke-sub-5babae40-nqjmg\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:18.008024 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.008000    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81255a89-fdlp5\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.012383 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.012363    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:18.012473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.012381    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:20:18.012473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.012402    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:20:18.031875 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.031849    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"]
Oct 30 19:20:18.109438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.109401    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81255a89-fdlp5\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.109640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.109530    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81255a89-fdlp5\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.109640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.109569    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5babae40-nqjmg\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:18.109640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.109590    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5babae40-nqjmg\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:18.109753 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.109727    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm78t\" (UniqueName: \"kubernetes.io/projected/45ffcbb7-526d-4a6c-842f-d7112cd0d942-kube-api-access-gm78t\") pod \"maas-api-revoke-sub-81255a89-fdlp5\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.109815 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.109796    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqcx\" (UniqueName: \"kubernetes.io/projected/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-kube-api-access-2vqcx\") pod \"maas-api-revoke-sub-5babae40-nqjmg\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:18.110346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.110319    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5babae40-nqjmg\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:18.110346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.110326    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81255a89-fdlp5\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.112026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.112007    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5babae40-nqjmg\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:18.112087 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.112014    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81255a89-fdlp5\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.166931 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.166858    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm78t\" (UniqueName: \"kubernetes.io/projected/45ffcbb7-526d-4a6c-842f-d7112cd0d942-kube-api-access-gm78t\") pod \"maas-api-revoke-sub-81255a89-fdlp5\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.166931 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.166862    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqcx\" (UniqueName: \"kubernetes.io/projected/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-kube-api-access-2vqcx\") pod \"maas-api-revoke-sub-5babae40-nqjmg\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:18.221935 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.221904    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:18.276854 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.276817    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:18.598354 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.598320    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"]
Oct 30 19:20:18.599646 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:20:18.599619    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227bb3dc_ffe9_4128_8ca3_ff4955cf8327.slice/crio-c27767f03fcaa4d4aaf54f98a99443d140082cde4ab60d5acb4b16dbb5f6a075 WatchSource:0}: Error finding container c27767f03fcaa4d4aaf54f98a99443d140082cde4ab60d5acb4b16dbb5f6a075: Status 404 returned error can't find the container with id c27767f03fcaa4d4aaf54f98a99443d140082cde4ab60d5acb4b16dbb5f6a075
Oct 30 19:20:18.707166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.707135    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"]
Oct 30 19:20:18.708676 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:20:18.708645    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ffcbb7_526d_4a6c_842f_d7112cd0d942.slice/crio-ddde2079fe1c83ad79152eddc1edcf9e7c4068db3cddbcf336680d83bad3d941 WatchSource:0}: Error finding container ddde2079fe1c83ad79152eddc1edcf9e7c4068db3cddbcf336680d83bad3d941: Status 404 returned error can't find the container with id ddde2079fe1c83ad79152eddc1edcf9e7c4068db3cddbcf336680d83bad3d941
Oct 30 19:20:18.746028 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.745992    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"]
Oct 30 19:20:18.750542 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.750513    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-611053-xjrr7"]
Oct 30 19:20:18.979733 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.979700    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5" event={"ID":"45ffcbb7-526d-4a6c-842f-d7112cd0d942","Type":"ContainerStarted","Data":"ddde2079fe1c83ad79152eddc1edcf9e7c4068db3cddbcf336680d83bad3d941"}
Oct 30 19:20:18.981269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.981238    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg" event={"ID":"227bb3dc-ffe9-4128-8ca3-ff4955cf8327","Type":"ContainerStarted","Data":"5a0a27d57015050f1f56f6436786beeed1e96783ae055584c980bb0741f68e25"}
Oct 30 19:20:18.981400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:18.981292    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg" event={"ID":"227bb3dc-ffe9-4128-8ca3-ff4955cf8327","Type":"ContainerStarted","Data":"c27767f03fcaa4d4aaf54f98a99443d140082cde4ab60d5acb4b16dbb5f6a075"}
Oct 30 19:20:19.097939 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:19.097883    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg" podStartSLOduration=2.097869456 podStartE2EDuration="2.097869456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:19.096438506 +0000 UTC m=+2509.984355594" watchObservedRunningTime="2026-10-30 19:20:19.097869456 +0000 UTC m=+2509.985786544"
Oct 30 19:20:19.219728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:19.219699    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba5ad96-fce2-445b-8763-8093185d58f5" path="/var/lib/kubelet/pods/9ba5ad96-fce2-445b-8763-8093185d58f5/volumes"
Oct 30 19:20:19.989133 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:19.989102    4267 generic.go:356] "Generic (PLEG): container finished" podID="45ffcbb7-526d-4a6c-842f-d7112cd0d942" containerID="5f645f22a658602a3bf6eb40483f2117a9f6c4282f117a71437660947ab8855e" exitCode=0
Oct 30 19:20:19.989611 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:19.989187    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5" event={"ID":"45ffcbb7-526d-4a6c-842f-d7112cd0d942","Type":"ContainerDied","Data":"5f645f22a658602a3bf6eb40483f2117a9f6c4282f117a71437660947ab8855e"}
Oct 30 19:20:19.990776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:19.990755    4267 generic.go:356] "Generic (PLEG): container finished" podID="227bb3dc-ffe9-4128-8ca3-ff4955cf8327" containerID="5a0a27d57015050f1f56f6436786beeed1e96783ae055584c980bb0741f68e25" exitCode=0
Oct 30 19:20:19.990895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:19.990837    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg" event={"ID":"227bb3dc-ffe9-4128-8ca3-ff4955cf8327","Type":"ContainerDied","Data":"5a0a27d57015050f1f56f6436786beeed1e96783ae055584c980bb0741f68e25"}
Oct 30 19:20:20.158518 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:20:20.158467    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:20:20.514564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.514525    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main" containerID="cri-o://ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8" gracePeriod=45
Oct 30 19:20:20.796214 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.796188    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:20:20.840266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.840232    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-model-cache\") pod \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") "
Oct 30 19:20:20.840445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.840325    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-dshm\") pod \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") "
Oct 30 19:20:20.840445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.840376    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-home\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-home\") pod \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") "
Oct 30 19:20:20.840445 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.840432    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tls-certs\") pod \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") "
Oct 30 19:20:20.840651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.840501    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-model-cache" pod "6e86f312-519f-4842-8e30-1c3a1aaf4ae7" (UID: "6e86f312-519f-4842-8e30-1c3a1aaf4ae7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:20.840651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.840516    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-kube-api-access-7c695\" (UniqueName: \"kubernetes.io/projected/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-kube-api-access-7c695\") pod \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") "
Oct 30 19:20:20.840764 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.840689    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-home" pod "6e86f312-519f-4842-8e30-1c3a1aaf4ae7" (UID: "6e86f312-519f-4842-8e30-1c3a1aaf4ae7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:20.840764 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.840713    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tmp-dir\") pod \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\" (UID: \"6e86f312-519f-4842-8e30-1c3a1aaf4ae7\") "
Oct 30 19:20:20.840969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.840945    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tmp-dir" pod "6e86f312-519f-4842-8e30-1c3a1aaf4ae7" (UID: "6e86f312-519f-4842-8e30-1c3a1aaf4ae7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:20.841192 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.841123    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:20.841192 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.841146    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:20.841192 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.841159    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:20.842886 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.842863    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tls-certs" pod "6e86f312-519f-4842-8e30-1c3a1aaf4ae7" (UID: "6e86f312-519f-4842-8e30-1c3a1aaf4ae7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:20.843018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.843000    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-kube-api-access-7c695" pod "6e86f312-519f-4842-8e30-1c3a1aaf4ae7" (UID: "6e86f312-519f-4842-8e30-1c3a1aaf4ae7"). InnerVolumeSpecName "kube-api-access-7c695". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:20.843077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.843030    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-dshm" pod "6e86f312-519f-4842-8e30-1c3a1aaf4ae7" (UID: "6e86f312-519f-4842-8e30-1c3a1aaf4ae7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:20.941976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.941934    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:20.941976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.941972    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:20.941976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:20.941982    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c695\" (UniqueName: \"kubernetes.io/projected/6e86f312-519f-4842-8e30-1c3a1aaf4ae7-kube-api-access-7c695\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:21.001150 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.001115    4267 generic.go:356] "Generic (PLEG): container finished" podID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerID="ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8" exitCode=0
Oct 30 19:20:21.001666 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.001165    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"
Oct 30 19:20:21.001666 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.001195    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" event={"ID":"6e86f312-519f-4842-8e30-1c3a1aaf4ae7","Type":"ContainerDied","Data":"ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8"}
Oct 30 19:20:21.001666 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.001255    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq" event={"ID":"6e86f312-519f-4842-8e30-1c3a1aaf4ae7","Type":"ContainerDied","Data":"3e5d63b99084fdb6d75e4303dfb605f48e631f731b56d969b3d60742af97ff78"}
Oct 30 19:20:21.001666 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.001278    4267 scope.go:122] "RemoveContainer" containerID="ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8"
Oct 30 19:20:21.014253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.014221    4267 scope.go:122] "RemoveContainer" containerID="ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8"
Oct 30 19:20:21.015210 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:20:21.015173    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8\": container with ID starting with ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8 not found: ID does not exist" containerID="ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8"
Oct 30 19:20:21.015326 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.015213    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8"} err="failed to get container status \"ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8\": rpc error: code = NotFound desc = could not find container \"ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8\": container with ID starting with ef45a49ed83fb7cefc5c3ce51e2c7eb246d01438dde65ce6af3d0f18a81e22a8 not found: ID does not exist"
Oct 30 19:20:21.142190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.142160    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:21.170572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.170531    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"]
Oct 30 19:20:21.179328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.175534    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bc98e1/distinct-w3-bc98e1-kserve-d489758db-4h8sq"]
Oct 30 19:20:21.183633 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.183604    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:21.220289 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.220256    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" path="/var/lib/kubelet/pods/6e86f312-519f-4842-8e30-1c3a1aaf4ae7/volumes"
Oct 30 19:20:21.245180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.245145    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ffcbb7-526d-4a6c-842f-d7112cd0d942-kube-api-access-gm78t\" (UniqueName: \"kubernetes.io/projected/45ffcbb7-526d-4a6c-842f-d7112cd0d942-kube-api-access-gm78t\") pod \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") "
Oct 30 19:20:21.245374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.245225    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-openshift-ca\") pod \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") "
Oct 30 19:20:21.245374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.245279    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-kube-api-access-2vqcx\" (UniqueName: \"kubernetes.io/projected/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-kube-api-access-2vqcx\") pod \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") "
Oct 30 19:20:21.245374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.245325    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-xks-ca\") pod \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\" (UID: \"45ffcbb7-526d-4a6c-842f-d7112cd0d942\") "
Oct 30 19:20:21.245572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.245410    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-openshift-ca\") pod \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") "
Oct 30 19:20:21.245572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.245438    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-xks-ca\") pod \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\" (UID: \"227bb3dc-ffe9-4128-8ca3-ff4955cf8327\") "
Oct 30 19:20:21.245758 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.245655    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-openshift-ca" pod "45ffcbb7-526d-4a6c-842f-d7112cd0d942" (UID: "45ffcbb7-526d-4a6c-842f-d7112cd0d942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:21.245884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.245851    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-openshift-ca" pod "227bb3dc-ffe9-4128-8ca3-ff4955cf8327" (UID: "227bb3dc-ffe9-4128-8ca3-ff4955cf8327"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:21.246027 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.245976    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:21.247525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.247460    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-xks-ca" pod "45ffcbb7-526d-4a6c-842f-d7112cd0d942" (UID: "45ffcbb7-526d-4a6c-842f-d7112cd0d942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:21.247638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.247608    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-kube-api-access-2vqcx" pod "227bb3dc-ffe9-4128-8ca3-ff4955cf8327" (UID: "227bb3dc-ffe9-4128-8ca3-ff4955cf8327"). InnerVolumeSpecName "kube-api-access-2vqcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:21.247685 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.247637    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-xks-ca" pod "227bb3dc-ffe9-4128-8ca3-ff4955cf8327" (UID: "227bb3dc-ffe9-4128-8ca3-ff4955cf8327"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:21.247685 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.247651    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ffcbb7-526d-4a6c-842f-d7112cd0d942-kube-api-access-gm78t" pod "45ffcbb7-526d-4a6c-842f-d7112cd0d942" (UID: "45ffcbb7-526d-4a6c-842f-d7112cd0d942"). InnerVolumeSpecName "kube-api-access-gm78t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:21.347566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.347444    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqcx\" (UniqueName: \"kubernetes.io/projected/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-kube-api-access-2vqcx\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:21.347566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.347479    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ffcbb7-526d-4a6c-842f-d7112cd0d942-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:21.347566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.347540    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:21.347566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.347555    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227bb3dc-ffe9-4128-8ca3-ff4955cf8327-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:21.347566 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:21.347564    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm78t\" (UniqueName: \"kubernetes.io/projected/45ffcbb7-526d-4a6c-842f-d7112cd0d942-kube-api-access-gm78t\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:22.008995 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:22.008963    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"
Oct 30 19:20:22.009467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:22.008993    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg" event={"ID":"227bb3dc-ffe9-4128-8ca3-ff4955cf8327","Type":"ContainerDied","Data":"c27767f03fcaa4d4aaf54f98a99443d140082cde4ab60d5acb4b16dbb5f6a075"}
Oct 30 19:20:22.009467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:22.009053    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c27767f03fcaa4d4aaf54f98a99443d140082cde4ab60d5acb4b16dbb5f6a075"
Oct 30 19:20:22.011631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:22.011606    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5" event={"ID":"45ffcbb7-526d-4a6c-842f-d7112cd0d942","Type":"ContainerDied","Data":"ddde2079fe1c83ad79152eddc1edcf9e7c4068db3cddbcf336680d83bad3d941"}
Oct 30 19:20:22.011748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:22.011650    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddde2079fe1c83ad79152eddc1edcf9e7c4068db3cddbcf336680d83bad3d941"
Oct 30 19:20:22.011748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:22.011651    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"
Oct 30 19:20:22.883505 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:22.883459    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m" podUID="53707cb2-f063-4273-bd83-5dc56d5e2c77" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:31.581130 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:31.581092    4267 scope.go:122] "RemoveContainer" containerID="dbe4b33145a6a3f5133173288e954df038c6d0e6aee70dfe119cca41f6071eb8"
Oct 30 19:20:31.592405 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:31.592379    4267 scope.go:122] "RemoveContainer" containerID="9594f6677158ded42b76e72947aac8d9e7274f56871e4db5ff893c626857514c"
Oct 30 19:20:32.883670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:32.883631    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m" podUID="53707cb2-f063-4273-bd83-5dc56d5e2c77" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:37.653953 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:37.653838    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 19:20:37.741676 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:37.741635    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53707cb2-f063-4273-bd83-5dc56d5e2c77-kube-api-access-bj55d\" (UniqueName: \"kubernetes.io/projected/53707cb2-f063-4273-bd83-5dc56d5e2c77-kube-api-access-bj55d\") pod \"53707cb2-f063-4273-bd83-5dc56d5e2c77\" (UID: \"53707cb2-f063-4273-bd83-5dc56d5e2c77\") "
Oct 30 19:20:37.741880 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:37.741832    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53707cb2-f063-4273-bd83-5dc56d5e2c77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53707cb2-f063-4273-bd83-5dc56d5e2c77-plugins-config-volume\") pod \"53707cb2-f063-4273-bd83-5dc56d5e2c77\" (UID: \"53707cb2-f063-4273-bd83-5dc56d5e2c77\") "
Oct 30 19:20:37.742209 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:37.742185    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53707cb2-f063-4273-bd83-5dc56d5e2c77-plugins-config-volume" pod "53707cb2-f063-4273-bd83-5dc56d5e2c77" (UID: "53707cb2-f063-4273-bd83-5dc56d5e2c77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:37.744357 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:37.744332    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53707cb2-f063-4273-bd83-5dc56d5e2c77-kube-api-access-bj55d" pod "53707cb2-f063-4273-bd83-5dc56d5e2c77" (UID: "53707cb2-f063-4273-bd83-5dc56d5e2c77"). InnerVolumeSpecName "kube-api-access-bj55d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:37.843601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:37.843561    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj55d\" (UniqueName: \"kubernetes.io/projected/53707cb2-f063-4273-bd83-5dc56d5e2c77-kube-api-access-bj55d\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:37.843601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:37.843602    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53707cb2-f063-4273-bd83-5dc56d5e2c77-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:38.123524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:38.123472    4267 generic.go:356] "Generic (PLEG): container finished" podID="53707cb2-f063-4273-bd83-5dc56d5e2c77" containerID="ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190" exitCode=0
Oct 30 19:20:38.123524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:38.123531    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m" event={"ID":"53707cb2-f063-4273-bd83-5dc56d5e2c77","Type":"ContainerDied","Data":"ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190"}
Oct 30 19:20:38.123791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:38.123559    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"
Oct 30 19:20:38.123791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:38.123597    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m" event={"ID":"53707cb2-f063-4273-bd83-5dc56d5e2c77","Type":"ContainerDied","Data":"fefbc9100aaf34c3b6698256842a85bbe7a90e941c249d444f51d7ab8f554a83"}
Oct 30 19:20:38.123791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:38.123618    4267 scope.go:122] "RemoveContainer" containerID="ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190"
Oct 30 19:20:38.134091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:38.134072    4267 scope.go:122] "RemoveContainer" containerID="ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190"
Oct 30 19:20:38.134348 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:20:38.134331    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190\": container with ID starting with ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190 not found: ID does not exist" containerID="ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190"
Oct 30 19:20:38.134426 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:38.134355    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190"} err="failed to get container status \"ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190\": rpc error: code = NotFound desc = could not find container \"ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190\": container with ID starting with ba13511f0b2f8596eded778071410d225d5d40dbd3dd6afb79eef31bc7a5d190 not found: ID does not exist"
Oct 30 19:20:38.292778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:38.292740    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"]
Oct 30 19:20:38.342292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:38.342258    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bc98e1-584578cd8c-gfv8m"]
Oct 30 19:20:39.218895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:39.218862    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53707cb2-f063-4273-bd83-5dc56d5e2c77" path="/var/lib/kubelet/pods/53707cb2-f063-4273-bd83-5dc56d5e2c77/volumes"
Oct 30 19:20:39.447035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:39.446997    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"]
Oct 30 19:20:39.494741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:39.494666    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bc98e1-x7gc4"]
Oct 30 19:20:41.049563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:41.049523    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"]
Oct 30 19:20:41.099232 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:41.099199    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"]
Oct 30 19:20:41.101798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:41.101784    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aef76865-9ghnx"]
Oct 30 19:20:41.103922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:41.103900    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c9c0534-4hlg5"]
Oct 30 19:20:41.219714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:41.219682    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea552f9-ef32-4d41-9d2d-5429fc3f2049" path="/var/lib/kubelet/pods/3ea552f9-ef32-4d41-9d2d-5429fc3f2049/volumes"
Oct 30 19:20:41.220133 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:41.220119    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7efeda72-a025-4e51-a54f-15ace119478f" path="/var/lib/kubelet/pods/7efeda72-a025-4e51-a54f-15ace119478f/volumes"
Oct 30 19:20:41.220568 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:41.220554    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a9ec66-fd5d-411e-a26e-03f17dae3787" path="/var/lib/kubelet/pods/94a9ec66-fd5d-411e-a26e-03f17dae3787/volumes"
Oct 30 19:20:44.588373 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588329    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"]
Oct 30 19:20:44.588801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588724    4267 cpu_manager.go:414] "containerMap: removing container" podUID="53707cb2-f063-4273-bd83-5dc56d5e2c77" containerName="payload-pre-processing"
Oct 30 19:20:44.588801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588736    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53707cb2-f063-4273-bd83-5dc56d5e2c77" containerName="payload-pre-processing"
Oct 30 19:20:44.588801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588763    4267 cpu_manager.go:414] "containerMap: removing container" podUID="227bb3dc-ffe9-4128-8ca3-ff4955cf8327" containerName="revoke-keys"
Oct 30 19:20:44.588801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588769    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227bb3dc-ffe9-4128-8ca3-ff4955cf8327" containerName="revoke-keys"
Oct 30 19:20:44.588801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588776    4267 cpu_manager.go:414] "containerMap: removing container" podUID="45ffcbb7-526d-4a6c-842f-d7112cd0d942" containerName="revoke-keys"
Oct 30 19:20:44.588801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588781    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ffcbb7-526d-4a6c-842f-d7112cd0d942" containerName="revoke-keys"
Oct 30 19:20:44.588801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588788    4267 cpu_manager.go:414] "containerMap: removing container" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main"
Oct 30 19:20:44.588801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588793    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main"
Oct 30 19:20:44.589055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588852    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="227bb3dc-ffe9-4128-8ca3-ff4955cf8327" containerName="revoke-keys"
Oct 30 19:20:44.589055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588859    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ffcbb7-526d-4a6c-842f-d7112cd0d942" containerName="revoke-keys"
Oct 30 19:20:44.589055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588865    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e86f312-519f-4842-8e30-1c3a1aaf4ae7" containerName="main"
Oct 30 19:20:44.589055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.588873    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="53707cb2-f063-4273-bd83-5dc56d5e2c77" containerName="payload-pre-processing"
Oct 30 19:20:44.593547 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.593529    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:44.711560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.711518    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c36105a9-6szp9\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:44.711764 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.711637    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp2r4\" (UniqueName: \"kubernetes.io/projected/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-kube-api-access-xp2r4\") pod \"maas-api-revoke-sub-c36105a9-6szp9\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:44.711764 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.711726    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c36105a9-6szp9\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:44.747252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.747220    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:44.747252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.747224    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:20:44.747547 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.747264    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:20:44.747547 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.747355    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:20:44.750577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.750552    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"]
Oct 30 19:20:44.812346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.812314    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c36105a9-6szp9\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:44.812585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.812372    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp2r4\" (UniqueName: \"kubernetes.io/projected/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-kube-api-access-xp2r4\") pod \"maas-api-revoke-sub-c36105a9-6szp9\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:44.812585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.812414    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c36105a9-6szp9\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:44.813221 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.813192    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c36105a9-6szp9\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:44.815821 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.815793    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c36105a9-6szp9\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:44.869399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.869320    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp2r4\" (UniqueName: \"kubernetes.io/projected/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-kube-api-access-xp2r4\") pod \"maas-api-revoke-sub-c36105a9-6szp9\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:44.906224 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:44.906188    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:45.338741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:45.338574    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"]
Oct 30 19:20:45.341529 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:20:45.341489    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d78c2e2_6d38_4a0e_988c_1276f4ebd544.slice/crio-786ccada24558e9cbfe2a8af6c19edfdb33b8d6fd2340a7c533564acfd30cb3d WatchSource:0}: Error finding container 786ccada24558e9cbfe2a8af6c19edfdb33b8d6fd2340a7c533564acfd30cb3d: Status 404 returned error can't find the container with id 786ccada24558e9cbfe2a8af6c19edfdb33b8d6fd2340a7c533564acfd30cb3d
Oct 30 19:20:46.178106 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:46.178075    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9" event={"ID":"4d78c2e2-6d38-4a0e-988c-1276f4ebd544","Type":"ContainerStarted","Data":"2b3bfe5149e4d6601c6c79433a1f5bee19b8c1ce661a1809b672afe86307298f"}
Oct 30 19:20:46.178417 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:46.178140    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9" event={"ID":"4d78c2e2-6d38-4a0e-988c-1276f4ebd544","Type":"ContainerStarted","Data":"786ccada24558e9cbfe2a8af6c19edfdb33b8d6fd2340a7c533564acfd30cb3d"}
Oct 30 19:20:46.972861 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:46.972812    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"]
Oct 30 19:20:46.973053 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:46.973032    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerName="istio-proxy" containerID="cri-o://70308c602f44297543e956a5a6a4f74e787c8a7215b2d502ef56bac88d608f27" gracePeriod=30
Oct 30 19:20:47.194306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.194266    4267 generic.go:356] "Generic (PLEG): container finished" podID="4d78c2e2-6d38-4a0e-988c-1276f4ebd544" containerID="2b3bfe5149e4d6601c6c79433a1f5bee19b8c1ce661a1809b672afe86307298f" exitCode=0
Oct 30 19:20:47.194788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.194340    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9" event={"ID":"4d78c2e2-6d38-4a0e-988c-1276f4ebd544","Type":"ContainerDied","Data":"2b3bfe5149e4d6601c6c79433a1f5bee19b8c1ce661a1809b672afe86307298f"}
Oct 30 19:20:47.333332 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.333307    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:47.438726 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.438692    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-xks-ca\") pod \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") "
Oct 30 19:20:47.438726 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.438729    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-openshift-ca\") pod \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") "
Oct 30 19:20:47.438969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.438785    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-kube-api-access-xp2r4\" (UniqueName: \"kubernetes.io/projected/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-kube-api-access-xp2r4\") pod \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\" (UID: \"4d78c2e2-6d38-4a0e-988c-1276f4ebd544\") "
Oct 30 19:20:47.439209 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.439182    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-openshift-ca" pod "4d78c2e2-6d38-4a0e-988c-1276f4ebd544" (UID: "4d78c2e2-6d38-4a0e-988c-1276f4ebd544"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:47.440922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.440898    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-kube-api-access-xp2r4" pod "4d78c2e2-6d38-4a0e-988c-1276f4ebd544" (UID: "4d78c2e2-6d38-4a0e-988c-1276f4ebd544"). InnerVolumeSpecName "kube-api-access-xp2r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:47.441036 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.440899    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-xks-ca" pod "4d78c2e2-6d38-4a0e-988c-1276f4ebd544" (UID: "4d78c2e2-6d38-4a0e-988c-1276f4ebd544"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:47.539780 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.539689    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp2r4\" (UniqueName: \"kubernetes.io/projected/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-kube-api-access-xp2r4\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:47.539780 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.539721    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:47.539780 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:47.539732    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d78c2e2-6d38-4a0e-988c-1276f4ebd544-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:48.201393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:48.201352    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9" event={"ID":"4d78c2e2-6d38-4a0e-988c-1276f4ebd544","Type":"ContainerDied","Data":"786ccada24558e9cbfe2a8af6c19edfdb33b8d6fd2340a7c533564acfd30cb3d"}
Oct 30 19:20:48.201393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:48.201388    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"
Oct 30 19:20:48.201862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:48.201417    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="786ccada24558e9cbfe2a8af6c19edfdb33b8d6fd2340a7c533564acfd30cb3d"
Oct 30 19:20:50.158727 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:20:50.158686    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:20:52.227824 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.227795    4267 generic.go:356] "Generic (PLEG): container finished" podID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerID="70308c602f44297543e956a5a6a4f74e787c8a7215b2d502ef56bac88d608f27" exitCode=0
Oct 30 19:20:52.228167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.227865    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" event={"ID":"dec59e1d-c104-4fca-9793-1a55b8d16680","Type":"ContainerDied","Data":"70308c602f44297543e956a5a6a4f74e787c8a7215b2d502ef56bac88d608f27"}
Oct 30 19:20:52.228167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.227917    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4" event={"ID":"dec59e1d-c104-4fca-9793-1a55b8d16680","Type":"ContainerDied","Data":"04142a8fdab4badf0350fef1c0031e95fac5d8f0099cc9a2a2718442e960d52a"}
Oct 30 19:20:52.228167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.227934    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04142a8fdab4badf0350fef1c0031e95fac5d8f0099cc9a2a2718442e960d52a"
Oct 30 19:20:52.237362 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.237340    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:20:52.389458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389365    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-podinfo\") pod \"dec59e1d-c104-4fca-9793-1a55b8d16680\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") "
Oct 30 19:20:52.389458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389424    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-certs\") pod \"dec59e1d-c104-4fca-9793-1a55b8d16680\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") "
Oct 30 19:20:52.389458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389447    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-kube-api-access-7dfcx\" (UniqueName: \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-kube-api-access-7dfcx\") pod \"dec59e1d-c104-4fca-9793-1a55b8d16680\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") "
Oct 30 19:20:52.389796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389464    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-token\" (UniqueName: \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-token\") pod \"dec59e1d-c104-4fca-9793-1a55b8d16680\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") "
Oct 30 19:20:52.389796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389588    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-data\") pod \"dec59e1d-c104-4fca-9793-1a55b8d16680\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") "
Oct 30 19:20:52.389796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389638    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-socket\") pod \"dec59e1d-c104-4fca-9793-1a55b8d16680\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") "
Oct 30 19:20:52.389796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389667    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-credential-socket\") pod \"dec59e1d-c104-4fca-9793-1a55b8d16680\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") "
Oct 30 19:20:52.389796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389709    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-envoy\") pod \"dec59e1d-c104-4fca-9793-1a55b8d16680\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") "
Oct 30 19:20:52.389796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389744    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec59e1d-c104-4fca-9793-1a55b8d16680-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec59e1d-c104-4fca-9793-1a55b8d16680-istiod-ca-cert\") pod \"dec59e1d-c104-4fca-9793-1a55b8d16680\" (UID: \"dec59e1d-c104-4fca-9793-1a55b8d16680\") "
Oct 30 19:20:52.390122 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389797    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-certs" pod "dec59e1d-c104-4fca-9793-1a55b8d16680" (UID: "dec59e1d-c104-4fca-9793-1a55b8d16680"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:52.390122 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.389955    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-data" pod "dec59e1d-c104-4fca-9793-1a55b8d16680" (UID: "dec59e1d-c104-4fca-9793-1a55b8d16680"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:52.390244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.390212    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-credential-socket" pod "dec59e1d-c104-4fca-9793-1a55b8d16680" (UID: "dec59e1d-c104-4fca-9793-1a55b8d16680"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:52.390316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.390270    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec59e1d-c104-4fca-9793-1a55b8d16680-istiod-ca-cert" pod "dec59e1d-c104-4fca-9793-1a55b8d16680" (UID: "dec59e1d-c104-4fca-9793-1a55b8d16680"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:52.390316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.390280    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:52.390316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.390304    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:52.390316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.390270    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-socket" pod "dec59e1d-c104-4fca-9793-1a55b8d16680" (UID: "dec59e1d-c104-4fca-9793-1a55b8d16680"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:52.392040 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.392011    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-token" pod "dec59e1d-c104-4fca-9793-1a55b8d16680" (UID: "dec59e1d-c104-4fca-9793-1a55b8d16680"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:52.392158 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.392034    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-envoy" pod "dec59e1d-c104-4fca-9793-1a55b8d16680" (UID: "dec59e1d-c104-4fca-9793-1a55b8d16680"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:52.392158 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.392116    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-podinfo" pod "dec59e1d-c104-4fca-9793-1a55b8d16680" (UID: "dec59e1d-c104-4fca-9793-1a55b8d16680"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:52.392158 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.392139    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-kube-api-access-7dfcx" pod "dec59e1d-c104-4fca-9793-1a55b8d16680" (UID: "dec59e1d-c104-4fca-9793-1a55b8d16680"). InnerVolumeSpecName "kube-api-access-7dfcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:52.491021 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.490989    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:52.491021 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.491019    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:52.491021 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.491032    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:52.491295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.491045    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec59e1d-c104-4fca-9793-1a55b8d16680-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:52.491295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.491058    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:52.491295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.491070    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfcx\" (UniqueName: \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-kube-api-access-7dfcx\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:52.491295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:52.491082    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec59e1d-c104-4fca-9793-1a55b8d16680-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:20:53.234072 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:53.234038    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"
Oct 30 19:20:53.449167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:53.449125    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"]
Oct 30 19:20:53.502052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:53.501952    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc98e1-openshift-default-5b4ff68f6c-hkxc4"]
Oct 30 19:20:55.219856 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:20:55.219819    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" path="/var/lib/kubelet/pods/dec59e1d-c104-4fca-9793-1a55b8d16680/volumes"
Oct 30 19:21:15.758932 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.758850    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"]
Oct 30 19:21:15.759387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.759229    4267 cpu_manager.go:414] "containerMap: removing container" podUID="4d78c2e2-6d38-4a0e-988c-1276f4ebd544" containerName="revoke-keys"
Oct 30 19:21:15.759387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.759241    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d78c2e2-6d38-4a0e-988c-1276f4ebd544" containerName="revoke-keys"
Oct 30 19:21:15.759387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.759256    4267 cpu_manager.go:414] "containerMap: removing container" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerName="istio-proxy"
Oct 30 19:21:15.759387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.759265    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerName="istio-proxy"
Oct 30 19:21:15.759387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.759348    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d78c2e2-6d38-4a0e-988c-1276f4ebd544" containerName="revoke-keys"
Oct 30 19:21:15.759387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.759356    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec59e1d-c104-4fca-9793-1a55b8d16680" containerName="istio-proxy"
Oct 30 19:21:15.762449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.762429    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:15.816453 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.816416    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:15.824065 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.824030    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvtk\" (UniqueName: \"kubernetes.io/projected/400aabd2-4d58-44f7-959f-aeb01952b4b3-kube-api-access-fpvtk\") pod \"maas-api-revoke-sub-8cae1c75-sm46n\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:15.824237 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.824112    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cae1c75-sm46n\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:15.824310 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.824245    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cae1c75-sm46n\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:15.866367 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.866325    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:21:15.866367 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.866339    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:21:15.866707 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.866676    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:21:15.869625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.869601    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"]
Oct 30 19:21:15.925723 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.925677    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cae1c75-sm46n\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:15.925927 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.925741    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvtk\" (UniqueName: \"kubernetes.io/projected/400aabd2-4d58-44f7-959f-aeb01952b4b3-kube-api-access-fpvtk\") pod \"maas-api-revoke-sub-8cae1c75-sm46n\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:15.925927 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.925845    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cae1c75-sm46n\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:15.926442 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.926416    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cae1c75-sm46n\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:15.928212 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.928186    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cae1c75-sm46n\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:15.982375 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:15.982341    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvtk\" (UniqueName: \"kubernetes.io/projected/400aabd2-4d58-44f7-959f-aeb01952b4b3-kube-api-access-fpvtk\") pod \"maas-api-revoke-sub-8cae1c75-sm46n\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:16.075670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:16.075589    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:16.447859 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:16.447823    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"]
Oct 30 19:21:16.448146 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:21:16.448125    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400aabd2_4d58_44f7_959f_aeb01952b4b3.slice/crio-afdbadb94acf7cd53ac9dd77b597b88b9209561d9e70b9945422341bf9af44d3 WatchSource:0}: Error finding container afdbadb94acf7cd53ac9dd77b597b88b9209561d9e70b9945422341bf9af44d3: Status 404 returned error can't find the container with id afdbadb94acf7cd53ac9dd77b597b88b9209561d9e70b9945422341bf9af44d3
Oct 30 19:21:17.382649 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:17.382615    4267 generic.go:356] "Generic (PLEG): container finished" podID="400aabd2-4d58-44f7-959f-aeb01952b4b3" containerID="536dd7f7697f75400f72cda01232b5b51d6abe61e4199a90699c7ce2a1c0282e" exitCode=0
Oct 30 19:21:17.383046 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:17.382703    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n" event={"ID":"400aabd2-4d58-44f7-959f-aeb01952b4b3","Type":"ContainerDied","Data":"536dd7f7697f75400f72cda01232b5b51d6abe61e4199a90699c7ce2a1c0282e"}
Oct 30 19:21:17.383046 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:17.382753    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n" event={"ID":"400aabd2-4d58-44f7-959f-aeb01952b4b3","Type":"ContainerStarted","Data":"afdbadb94acf7cd53ac9dd77b597b88b9209561d9e70b9945422341bf9af44d3"}
Oct 30 19:21:18.530641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:18.530614    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:18.652020 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:18.651943    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-xks-ca\") pod \"400aabd2-4d58-44f7-959f-aeb01952b4b3\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") "
Oct 30 19:21:18.652206 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:18.652061    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-openshift-ca\") pod \"400aabd2-4d58-44f7-959f-aeb01952b4b3\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") "
Oct 30 19:21:18.652206 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:18.652122    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400aabd2-4d58-44f7-959f-aeb01952b4b3-kube-api-access-fpvtk\" (UniqueName: \"kubernetes.io/projected/400aabd2-4d58-44f7-959f-aeb01952b4b3-kube-api-access-fpvtk\") pod \"400aabd2-4d58-44f7-959f-aeb01952b4b3\" (UID: \"400aabd2-4d58-44f7-959f-aeb01952b4b3\") "
Oct 30 19:21:18.652760 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:18.652485    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-openshift-ca" pod "400aabd2-4d58-44f7-959f-aeb01952b4b3" (UID: "400aabd2-4d58-44f7-959f-aeb01952b4b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:18.654269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:18.654246    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-xks-ca" pod "400aabd2-4d58-44f7-959f-aeb01952b4b3" (UID: "400aabd2-4d58-44f7-959f-aeb01952b4b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:18.654386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:18.654344    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400aabd2-4d58-44f7-959f-aeb01952b4b3-kube-api-access-fpvtk" pod "400aabd2-4d58-44f7-959f-aeb01952b4b3" (UID: "400aabd2-4d58-44f7-959f-aeb01952b4b3"). InnerVolumeSpecName "kube-api-access-fpvtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:18.753512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:18.753460    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvtk\" (UniqueName: \"kubernetes.io/projected/400aabd2-4d58-44f7-959f-aeb01952b4b3-kube-api-access-fpvtk\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:18.753512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:18.753489    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:18.753719 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:18.753524    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400aabd2-4d58-44f7-959f-aeb01952b4b3-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.396110 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:19.396077    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"
Oct 30 19:21:19.396289 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:19.396119    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n" event={"ID":"400aabd2-4d58-44f7-959f-aeb01952b4b3","Type":"ContainerDied","Data":"afdbadb94acf7cd53ac9dd77b597b88b9209561d9e70b9945422341bf9af44d3"}
Oct 30 19:21:19.396289 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:19.396191    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afdbadb94acf7cd53ac9dd77b597b88b9209561d9e70b9945422341bf9af44d3"
Oct 30 19:21:20.158652 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:21:20.158610    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:21:31.638104 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:31.638068    4267 scope.go:122] "RemoveContainer" containerID="70308c602f44297543e956a5a6a4f74e787c8a7215b2d502ef56bac88d608f27"
Oct 30 19:21:31.647880 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:31.647858    4267 scope.go:122] "RemoveContainer" containerID="150cbab8332b37e077451d55bce0e5118d40c00cd011a6cf65efa2a48f0571f8"
Oct 30 19:21:43.588279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.588244    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"]
Oct 30 19:21:43.588908 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.588813    4267 cpu_manager.go:414] "containerMap: removing container" podUID="400aabd2-4d58-44f7-959f-aeb01952b4b3" containerName="revoke-keys"
Oct 30 19:21:43.588908 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.588835    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400aabd2-4d58-44f7-959f-aeb01952b4b3" containerName="revoke-keys"
Oct 30 19:21:43.589042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.588965    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="400aabd2-4d58-44f7-959f-aeb01952b4b3" containerName="revoke-keys"
Oct 30 19:21:43.593402 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.593380    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:43.654352 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.654309    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"]
Oct 30 19:21:43.695657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.695613    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:43.695889 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.695863    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:21:43.695964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.695764    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:21:43.695964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.695685    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:21:43.710012 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.709976    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xml7h\" (UniqueName: \"kubernetes.io/projected/ac8acad7-6dc3-4896-8fd0-ccc86febd863-kube-api-access-xml7h\") pod \"maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:43.710178 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.710083    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:43.710178 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.710129    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:43.811463 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.811422    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:43.811682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.811481    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:43.811682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.811572    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xml7h\" (UniqueName: \"kubernetes.io/projected/ac8acad7-6dc3-4896-8fd0-ccc86febd863-kube-api-access-xml7h\") pod \"maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:43.812115 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.812091    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:43.813884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.813859    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:43.868787 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.868710    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xml7h\" (UniqueName: \"kubernetes.io/projected/ac8acad7-6dc3-4896-8fd0-ccc86febd863-kube-api-access-xml7h\") pod \"maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:43.905720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:43.905689    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:44.278773 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:21:44.278738    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8acad7_6dc3_4896_8fd0_ccc86febd863.slice/crio-6958d95bf7409ba364c3957b2e8aeb08f7d59fe0e1433907c3051450b014371d WatchSource:0}: Error finding container 6958d95bf7409ba364c3957b2e8aeb08f7d59fe0e1433907c3051450b014371d: Status 404 returned error can't find the container with id 6958d95bf7409ba364c3957b2e8aeb08f7d59fe0e1433907c3051450b014371d
Oct 30 19:21:44.279377 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:44.279344    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"]
Oct 30 19:21:44.562059 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:44.561969    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m" event={"ID":"ac8acad7-6dc3-4896-8fd0-ccc86febd863","Type":"ContainerStarted","Data":"6958d95bf7409ba364c3957b2e8aeb08f7d59fe0e1433907c3051450b014371d"}
Oct 30 19:21:45.570317 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:45.570232    4267 generic.go:356] "Generic (PLEG): container finished" podID="ac8acad7-6dc3-4896-8fd0-ccc86febd863" containerID="9c6e8ce136c5c038438aa149925237e08c5c61de2d627f2ff9ce2a0ed3c3bbb0" exitCode=0
Oct 30 19:21:45.570727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:45.570316    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m" event={"ID":"ac8acad7-6dc3-4896-8fd0-ccc86febd863","Type":"ContainerDied","Data":"9c6e8ce136c5c038438aa149925237e08c5c61de2d627f2ff9ce2a0ed3c3bbb0"}
Oct 30 19:21:46.705001 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:46.704979    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:46.742703 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:46.742666    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-openshift-ca\") pod \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") "
Oct 30 19:21:46.742888 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:46.742721    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8acad7-6dc3-4896-8fd0-ccc86febd863-kube-api-access-xml7h\" (UniqueName: \"kubernetes.io/projected/ac8acad7-6dc3-4896-8fd0-ccc86febd863-kube-api-access-xml7h\") pod \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") "
Oct 30 19:21:46.742888 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:46.742842    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-xks-ca\") pod \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\" (UID: \"ac8acad7-6dc3-4896-8fd0-ccc86febd863\") "
Oct 30 19:21:46.743106 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:46.743079    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-openshift-ca" pod "ac8acad7-6dc3-4896-8fd0-ccc86febd863" (UID: "ac8acad7-6dc3-4896-8fd0-ccc86febd863"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:46.744944 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:46.744914    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-xks-ca" pod "ac8acad7-6dc3-4896-8fd0-ccc86febd863" (UID: "ac8acad7-6dc3-4896-8fd0-ccc86febd863"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:46.745060 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:46.744946    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8acad7-6dc3-4896-8fd0-ccc86febd863-kube-api-access-xml7h" pod "ac8acad7-6dc3-4896-8fd0-ccc86febd863" (UID: "ac8acad7-6dc3-4896-8fd0-ccc86febd863"). InnerVolumeSpecName "kube-api-access-xml7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:46.844740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:46.844659    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:46.844740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:46.844695    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xml7h\" (UniqueName: \"kubernetes.io/projected/ac8acad7-6dc3-4896-8fd0-ccc86febd863-kube-api-access-xml7h\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:46.844740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:46.844713    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8acad7-6dc3-4896-8fd0-ccc86febd863-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:21:47.583282 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:47.583195    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"
Oct 30 19:21:47.583282 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:47.583230    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m" event={"ID":"ac8acad7-6dc3-4896-8fd0-ccc86febd863","Type":"ContainerDied","Data":"6958d95bf7409ba364c3957b2e8aeb08f7d59fe0e1433907c3051450b014371d"}
Oct 30 19:21:47.583282 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:47.583279    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6958d95bf7409ba364c3957b2e8aeb08f7d59fe0e1433907c3051450b014371d"
Oct 30 19:21:50.158176 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:21:50.158134    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:21:59.852955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:59.852913    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"]
Oct 30 19:21:59.901501 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:21:59.901463    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0fc1f-b8l5m"]
Oct 30 19:22:01.220724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:01.220693    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8acad7-6dc3-4896-8fd0-ccc86febd863" path="/var/lib/kubelet/pods/ac8acad7-6dc3-4896-8fd0-ccc86febd863/volumes"
Oct 30 19:22:08.115093 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.115051    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"]
Oct 30 19:22:08.115548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.115446    4267 cpu_manager.go:414] "containerMap: removing container" podUID="ac8acad7-6dc3-4896-8fd0-ccc86febd863" containerName="revoke-keys"
Oct 30 19:22:08.115548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.115458    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8acad7-6dc3-4896-8fd0-ccc86febd863" containerName="revoke-keys"
Oct 30 19:22:08.115548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.115542    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8acad7-6dc3-4896-8fd0-ccc86febd863" containerName="revoke-keys"
Oct 30 19:22:08.118846 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.118826    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:08.177214 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.177172    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"]
Oct 30 19:22:08.223336 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.223304    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:08.223665 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.223644    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:22:08.223665 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.223651    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:22:08.223810 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.223673    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:22:08.287162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.287126    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:08.287353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.287182    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqvqb\" (UniqueName: \"kubernetes.io/projected/9d0c2985-ad46-4101-be84-ae3ac748f560-kube-api-access-lqvqb\") pod \"maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:08.287353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.287315    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:08.388921 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.388826    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:08.388921 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.388896    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvqb\" (UniqueName: \"kubernetes.io/projected/9d0c2985-ad46-4101-be84-ae3ac748f560-kube-api-access-lqvqb\") pod \"maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:08.389176 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.388965    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:08.389593 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.389570    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:08.391328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.391306    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:08.447443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.447405    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvqb\" (UniqueName: \"kubernetes.io/projected/9d0c2985-ad46-4101-be84-ae3ac748f560-kube-api-access-lqvqb\") pod \"maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:08.731857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:08.731818    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:09.105416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:09.105384    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"]
Oct 30 19:22:09.106699 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:22:09.106657    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0c2985_ad46_4101_be84_ae3ac748f560.slice/crio-f6ed6bec92865b4eb4ff6eef0d86e0448d8231a59713a0cbd0e63f8fa84b6f7b WatchSource:0}: Error finding container f6ed6bec92865b4eb4ff6eef0d86e0448d8231a59713a0cbd0e63f8fa84b6f7b: Status 404 returned error can't find the container with id f6ed6bec92865b4eb4ff6eef0d86e0448d8231a59713a0cbd0e63f8fa84b6f7b
Oct 30 19:22:09.721088 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:09.721044    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs" event={"ID":"9d0c2985-ad46-4101-be84-ae3ac748f560","Type":"ContainerStarted","Data":"d061cd40c4924fbe679052e25fa39c1985c4f1d79ba17d1368d7df4911869b21"}
Oct 30 19:22:09.721664 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:09.721112    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs" event={"ID":"9d0c2985-ad46-4101-be84-ae3ac748f560","Type":"ContainerStarted","Data":"f6ed6bec92865b4eb4ff6eef0d86e0448d8231a59713a0cbd0e63f8fa84b6f7b"}
Oct 30 19:22:09.836469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:09.836411    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs" podStartSLOduration=1.836393434 podStartE2EDuration="1.836393434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:09.835424467 +0000 UTC m=+2620.723341557" watchObservedRunningTime="2026-10-30 19:22:09.836393434 +0000 UTC m=+2620.724310527"
Oct 30 19:22:10.729807 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:10.729765    4267 generic.go:356] "Generic (PLEG): container finished" podID="9d0c2985-ad46-4101-be84-ae3ac748f560" containerID="d061cd40c4924fbe679052e25fa39c1985c4f1d79ba17d1368d7df4911869b21" exitCode=0
Oct 30 19:22:10.730244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:10.729848    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs" event={"ID":"9d0c2985-ad46-4101-be84-ae3ac748f560","Type":"ContainerDied","Data":"d061cd40c4924fbe679052e25fa39c1985c4f1d79ba17d1368d7df4911869b21"}
Oct 30 19:22:11.884308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:11.884279    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:11.923781 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:11.923741    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0c2985-ad46-4101-be84-ae3ac748f560-kube-api-access-lqvqb\" (UniqueName: \"kubernetes.io/projected/9d0c2985-ad46-4101-be84-ae3ac748f560-kube-api-access-lqvqb\") pod \"9d0c2985-ad46-4101-be84-ae3ac748f560\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") "
Oct 30 19:22:11.923999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:11.923913    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-xks-ca\") pod \"9d0c2985-ad46-4101-be84-ae3ac748f560\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") "
Oct 30 19:22:11.923999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:11.923968    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-openshift-ca\") pod \"9d0c2985-ad46-4101-be84-ae3ac748f560\" (UID: \"9d0c2985-ad46-4101-be84-ae3ac748f560\") "
Oct 30 19:22:11.924386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:11.924361    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-openshift-ca" pod "9d0c2985-ad46-4101-be84-ae3ac748f560" (UID: "9d0c2985-ad46-4101-be84-ae3ac748f560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:11.925893 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:11.925862    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0c2985-ad46-4101-be84-ae3ac748f560-kube-api-access-lqvqb" pod "9d0c2985-ad46-4101-be84-ae3ac748f560" (UID: "9d0c2985-ad46-4101-be84-ae3ac748f560"). InnerVolumeSpecName "kube-api-access-lqvqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:11.926020 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:11.925978    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-xks-ca" pod "9d0c2985-ad46-4101-be84-ae3ac748f560" (UID: "9d0c2985-ad46-4101-be84-ae3ac748f560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:12.025822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:12.025731    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:12.025822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:12.025768    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0c2985-ad46-4101-be84-ae3ac748f560-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:12.025822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:12.025780    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqvqb\" (UniqueName: \"kubernetes.io/projected/9d0c2985-ad46-4101-be84-ae3ac748f560-kube-api-access-lqvqb\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:12.745560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:12.745524    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"
Oct 30 19:22:12.745560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:12.745549    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs" event={"ID":"9d0c2985-ad46-4101-be84-ae3ac748f560","Type":"ContainerDied","Data":"f6ed6bec92865b4eb4ff6eef0d86e0448d8231a59713a0cbd0e63f8fa84b6f7b"}
Oct 30 19:22:12.745804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:12.745608    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6ed6bec92865b4eb4ff6eef0d86e0448d8231a59713a0cbd0e63f8fa84b6f7b"
Oct 30 19:22:20.158207 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:22:20.158166    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:22:27.281824 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:27.281784    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"]
Oct 30 19:22:27.330182 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:27.330150    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-714b2c-v8gbs"]
Oct 30 19:22:29.219320 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:29.219275    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0c2985-ad46-4101-be84-ae3ac748f560" path="/var/lib/kubelet/pods/9d0c2985-ad46-4101-be84-ae3ac748f560/volumes"
Oct 30 19:22:31.679473 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:31.679440    4267 scope.go:122] "RemoveContainer" containerID="73a5020ddc79ecd6b4c4b85bf5d794df85ed320c6b40092105226ffdb6992620"
Oct 30 19:22:31.688784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:31.688761    4267 scope.go:122] "RemoveContainer" containerID="2b3748eca58bd46b8a08d99700a7ad1df2db5c7eb1d94703e6c5f0252df41813"
Oct 30 19:22:31.697226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:31.697205    4267 scope.go:122] "RemoveContainer" containerID="7b822f2742f26cb431563502beff4324c27db883473af00e82f5661756d6bd2f"
Oct 30 19:22:31.705421 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:31.705400    4267 scope.go:122] "RemoveContainer" containerID="dc5deb38742c79d334c74216fa0c7e167069a279fa3ccb1893d75cd7a2cbf927"
Oct 30 19:22:31.713839 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:31.713816    4267 scope.go:122] "RemoveContainer" containerID="039eb835bea300c76fdb4d9ccd9b04ba7177578c1179f4b50b87b2550d5aab75"
Oct 30 19:22:31.723838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:31.723814    4267 scope.go:122] "RemoveContainer" containerID="8cd08142e9fac4b2fcc519e69b15668595259eaa3bba17c0ef19e2347e3704b6"
Oct 30 19:22:40.493303 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:40.493267    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"]
Oct 30 19:22:47.305447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:47.305358    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"]
Oct 30 19:22:50.157865 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:22:50.157824    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:22:50.826937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.826900    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"]
Oct 30 19:22:50.827387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.827367    4267 cpu_manager.go:414] "containerMap: removing container" podUID="9d0c2985-ad46-4101-be84-ae3ac748f560" containerName="revoke-keys"
Oct 30 19:22:50.827523 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.827390    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0c2985-ad46-4101-be84-ae3ac748f560" containerName="revoke-keys"
Oct 30 19:22:50.827600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.827553    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0c2985-ad46-4101-be84-ae3ac748f560" containerName="revoke-keys"
Oct 30 19:22:50.832544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.832522    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:50.887728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.887682    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"]
Oct 30 19:22:50.933469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.933432    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59199a5a-qpv2d\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:50.933469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.933467    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsmkr\" (UniqueName: \"kubernetes.io/projected/8a1b4aa0-90f3-4b8b-a627-c0619805940f-kube-api-access-rsmkr\") pod \"maas-api-revoke-sub-59199a5a-qpv2d\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:50.933698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.933507    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59199a5a-qpv2d\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:50.934286 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.934265    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:22:50.934411 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.934387    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:50.934465 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.934395    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:22:50.934465 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:50.934394    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:22:51.034386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:51.034348    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59199a5a-qpv2d\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:51.034386 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:51.034388    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmkr\" (UniqueName: \"kubernetes.io/projected/8a1b4aa0-90f3-4b8b-a627-c0619805940f-kube-api-access-rsmkr\") pod \"maas-api-revoke-sub-59199a5a-qpv2d\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:51.034639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:51.034415    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59199a5a-qpv2d\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:51.035171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:51.035144    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59199a5a-qpv2d\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:51.036822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:51.036793    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59199a5a-qpv2d\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:51.091626 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:51.091555    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmkr\" (UniqueName: \"kubernetes.io/projected/8a1b4aa0-90f3-4b8b-a627-c0619805940f-kube-api-access-rsmkr\") pod \"maas-api-revoke-sub-59199a5a-qpv2d\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:51.146255 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:51.146223    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:51.524999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:51.524966    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"]
Oct 30 19:22:51.526137 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:22:51.526106    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1b4aa0_90f3_4b8b_a627_c0619805940f.slice/crio-65e34e0b4e2a7427c0df24996406a742f6c66e3c996c7221efba4e31344abe70 WatchSource:0}: Error finding container 65e34e0b4e2a7427c0df24996406a742f6c66e3c996c7221efba4e31344abe70: Status 404 returned error can't find the container with id 65e34e0b4e2a7427c0df24996406a742f6c66e3c996c7221efba4e31344abe70
Oct 30 19:22:51.999539 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:51.999484    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d" event={"ID":"8a1b4aa0-90f3-4b8b-a627-c0619805940f","Type":"ContainerStarted","Data":"587064e8e147c0182c0b8f8a5432eb29646d765864220b85f35e72065d756e0a"}
Oct 30 19:22:51.999721 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:51.999599    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d" event={"ID":"8a1b4aa0-90f3-4b8b-a627-c0619805940f","Type":"ContainerStarted","Data":"65e34e0b4e2a7427c0df24996406a742f6c66e3c996c7221efba4e31344abe70"}
Oct 30 19:22:52.114737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:52.114515    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d" podStartSLOduration=2.114484803 podStartE2EDuration="2.114484803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:52.113635296 +0000 UTC m=+2663.001552388" watchObservedRunningTime="2026-10-30 19:22:52.114484803 +0000 UTC m=+2663.002401893"
Oct 30 19:22:52.306452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:52.306366    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd" podUID="b52af405-150a-4af8-9bde-a9afad214d47" containerName="payload-pre-processing" containerID="cri-o://83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4" gracePeriod=25
Oct 30 19:22:53.006675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:53.006637    4267 generic.go:356] "Generic (PLEG): container finished" podID="8a1b4aa0-90f3-4b8b-a627-c0619805940f" containerID="587064e8e147c0182c0b8f8a5432eb29646d765864220b85f35e72065d756e0a" exitCode=0
Oct 30 19:22:53.007085 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:53.006727    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d" event={"ID":"8a1b4aa0-90f3-4b8b-a627-c0619805940f","Type":"ContainerDied","Data":"587064e8e147c0182c0b8f8a5432eb29646d765864220b85f35e72065d756e0a"}
Oct 30 19:22:53.048874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:53.048837    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"]
Oct 30 19:22:53.052936 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:53.052904    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a894cb-chmzx"]
Oct 30 19:22:53.218846 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:53.218810    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1162287-d097-4762-a3b6-28e28c7910e1" path="/var/lib/kubelet/pods/c1162287-d097-4762-a3b6-28e28c7910e1/volumes"
Oct 30 19:22:54.048955 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.048919    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"]
Oct 30 19:22:54.099062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.099020    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2cfd3f-9w7nb"]
Oct 30 19:22:54.148236 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.148213    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:54.266307 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.266232    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-openshift-ca\") pod \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") "
Oct 30 19:22:54.266472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.266316    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1b4aa0-90f3-4b8b-a627-c0619805940f-kube-api-access-rsmkr\" (UniqueName: \"kubernetes.io/projected/8a1b4aa0-90f3-4b8b-a627-c0619805940f-kube-api-access-rsmkr\") pod \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") "
Oct 30 19:22:54.266472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.266396    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-xks-ca\") pod \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\" (UID: \"8a1b4aa0-90f3-4b8b-a627-c0619805940f\") "
Oct 30 19:22:54.266791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.266660    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-openshift-ca" pod "8a1b4aa0-90f3-4b8b-a627-c0619805940f" (UID: "8a1b4aa0-90f3-4b8b-a627-c0619805940f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:54.266919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.266829    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:54.269058 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.269035    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-xks-ca" pod "8a1b4aa0-90f3-4b8b-a627-c0619805940f" (UID: "8a1b4aa0-90f3-4b8b-a627-c0619805940f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:54.269591 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.269575    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1b4aa0-90f3-4b8b-a627-c0619805940f-kube-api-access-rsmkr" pod "8a1b4aa0-90f3-4b8b-a627-c0619805940f" (UID: "8a1b4aa0-90f3-4b8b-a627-c0619805940f"). InnerVolumeSpecName "kube-api-access-rsmkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:54.367762 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.367731    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsmkr\" (UniqueName: \"kubernetes.io/projected/8a1b4aa0-90f3-4b8b-a627-c0619805940f-kube-api-access-rsmkr\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:54.367762 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:54.367759    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a1b4aa0-90f3-4b8b-a627-c0619805940f-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:55.019000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.018965    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d" event={"ID":"8a1b4aa0-90f3-4b8b-a627-c0619805940f","Type":"ContainerDied","Data":"65e34e0b4e2a7427c0df24996406a742f6c66e3c996c7221efba4e31344abe70"}
Oct 30 19:22:55.019000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.018988    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"
Oct 30 19:22:55.019295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.019013    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e34e0b4e2a7427c0df24996406a742f6c66e3c996c7221efba4e31344abe70"
Oct 30 19:22:55.219114 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.219081    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662447d4-f16d-458e-ba57-ce7cc87bcc8c" path="/var/lib/kubelet/pods/662447d4-f16d-458e-ba57-ce7cc87bcc8c/volumes"
Oct 30 19:22:55.513603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.513564    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main" containerID="cri-o://8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f" gracePeriod=45
Oct 30 19:22:55.765227 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.765161    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:22:55.881129 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881088    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a511f699-9b06-4893-a9f7-acb661ceb6c0-kube-api-access-hggq6\" (UniqueName: \"kubernetes.io/projected/a511f699-9b06-4893-a9f7-acb661ceb6c0-kube-api-access-hggq6\") pod \"a511f699-9b06-4893-a9f7-acb661ceb6c0\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") "
Oct 30 19:22:55.881335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881148    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-tmp-dir\") pod \"a511f699-9b06-4893-a9f7-acb661ceb6c0\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") "
Oct 30 19:22:55.881335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881230    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-model-cache\") pod \"a511f699-9b06-4893-a9f7-acb661ceb6c0\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") "
Oct 30 19:22:55.881335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881287    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-home\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-home\") pod \"a511f699-9b06-4893-a9f7-acb661ceb6c0\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") "
Oct 30 19:22:55.881476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881374    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-dshm\") pod \"a511f699-9b06-4893-a9f7-acb661ceb6c0\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") "
Oct 30 19:22:55.881476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881406    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a511f699-9b06-4893-a9f7-acb661ceb6c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a511f699-9b06-4893-a9f7-acb661ceb6c0-tls-certs\") pod \"a511f699-9b06-4893-a9f7-acb661ceb6c0\" (UID: \"a511f699-9b06-4893-a9f7-acb661ceb6c0\") "
Oct 30 19:22:55.881599 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881481    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-tmp-dir" pod "a511f699-9b06-4893-a9f7-acb661ceb6c0" (UID: "a511f699-9b06-4893-a9f7-acb661ceb6c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.881759 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881708    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-home" pod "a511f699-9b06-4893-a9f7-acb661ceb6c0" (UID: "a511f699-9b06-4893-a9f7-acb661ceb6c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.881906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881761    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-model-cache" pod "a511f699-9b06-4893-a9f7-acb661ceb6c0" (UID: "a511f699-9b06-4893-a9f7-acb661ceb6c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.881906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881807    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:55.881906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.881825    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:55.883535 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.883507    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a511f699-9b06-4893-a9f7-acb661ceb6c0-kube-api-access-hggq6" pod "a511f699-9b06-4893-a9f7-acb661ceb6c0" (UID: "a511f699-9b06-4893-a9f7-acb661ceb6c0"). InnerVolumeSpecName "kube-api-access-hggq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:55.883666 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.883569    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-dshm" pod "a511f699-9b06-4893-a9f7-acb661ceb6c0" (UID: "a511f699-9b06-4893-a9f7-acb661ceb6c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:55.883732 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.883716    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a511f699-9b06-4893-a9f7-acb661ceb6c0-tls-certs" pod "a511f699-9b06-4893-a9f7-acb661ceb6c0" (UID: "a511f699-9b06-4893-a9f7-acb661ceb6c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:55.982737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.982699    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:55.982737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.982733    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a511f699-9b06-4893-a9f7-acb661ceb6c0-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:55.982737 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.982744    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hggq6\" (UniqueName: \"kubernetes.io/projected/a511f699-9b06-4893-a9f7-acb661ceb6c0-kube-api-access-hggq6\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:55.983053 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:55.982754    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a511f699-9b06-4893-a9f7-acb661ceb6c0-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:56.027754 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:56.027657    4267 generic.go:356] "Generic (PLEG): container finished" podID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerID="8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f" exitCode=0
Oct 30 19:22:56.027754 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:56.027705    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"
Oct 30 19:22:56.027754 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:56.027726    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" event={"ID":"a511f699-9b06-4893-a9f7-acb661ceb6c0","Type":"ContainerDied","Data":"8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f"}
Oct 30 19:22:56.027993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:56.027774    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc" event={"ID":"a511f699-9b06-4893-a9f7-acb661ceb6c0","Type":"ContainerDied","Data":"f1b984557a69b546ce4f4c6f212349e34644b5a4ae58dd164c8114406f2a36a3"}
Oct 30 19:22:56.027993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:56.027796    4267 scope.go:122] "RemoveContainer" containerID="8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f"
Oct 30 19:22:56.038054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:56.038037    4267 scope.go:122] "RemoveContainer" containerID="8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f"
Oct 30 19:22:56.038338 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:22:56.038316    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f\": container with ID starting with 8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f not found: ID does not exist" containerID="8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f"
Oct 30 19:22:56.038438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:56.038344    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f"} err="failed to get container status \"8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f\": rpc error: code = NotFound desc = could not find container \"8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f\": container with ID starting with 8ec323161c505ef0b0aad3aa833f9cbccf35c31b6dcf062ebad250ec3981d93f not found: ID does not exist"
Oct 30 19:22:56.194483 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:56.194441    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"]
Oct 30 19:22:56.203394 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:56.199738    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-66bdc0/simulator-w2-66bdc0-kserve-7cb65cf86b-st4sc"]
Oct 30 19:22:57.222077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:57.222035    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" path="/var/lib/kubelet/pods/a511f699-9b06-4893-a9f7-acb661ceb6c0/volumes"
Oct 30 19:22:58.597025 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.596985    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"]
Oct 30 19:22:58.597487 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.597190    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq" podUID="b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" containerName="discovery" containerID="cri-o://40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45" gracePeriod=30
Oct 30 19:22:58.794098 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.794052    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd" podUID="b52af405-150a-4af8-9bde-a9afad214d47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:22:58.860456 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.860384    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:22:58.912216 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.912183    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-cacerts\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-cacerts\") pod \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") "
Oct 30 19:22:58.912402 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.912231    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-local-certs\") pod \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") "
Oct 30 19:22:58.912402 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.912272    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-kube-api-access-7cxf4\" (UniqueName: \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-kube-api-access-7cxf4\") pod \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") "
Oct 30 19:22:58.912402 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.912324    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-token\" (UniqueName: \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-token\") pod \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") "
Oct 30 19:22:58.912640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.912453    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-kubeconfig\") pod \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") "
Oct 30 19:22:58.912640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.912506    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-ca-configmap\") pod \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") "
Oct 30 19:22:58.912640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.912548    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-dns-cert\") pod \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\" (UID: \"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908\") "
Oct 30 19:22:58.913083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.913039    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-ca-configmap" pod "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" (UID: "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:58.914911 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.914883    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-cacerts" pod "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" (UID: "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:58.915141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.915115    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-token" pod "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" (UID: "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:58.915472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.915429    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-local-certs" pod "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" (UID: "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:58.915656 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.915628    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-dns-cert" pod "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" (UID: "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:58.915656 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.915639    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-kube-api-access-7cxf4" pod "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" (UID: "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908"). InnerVolumeSpecName "kube-api-access-7cxf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:58.915804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:58.915654    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-kubeconfig" pod "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" (UID: "b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:59.013642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.013599    4267 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-cacerts\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:59.013642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.013633    4267 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-local-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:59.013642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.013644    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cxf4\" (UniqueName: \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-kube-api-access-7cxf4\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:59.013642 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.013654    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:59.013945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.013663    4267 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-kubeconfig\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:59.013945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.013673    4267 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-ca-configmap\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:59.013945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.013682    4267 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908-istio-csr-dns-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:22:59.047137 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.047097    4267 generic.go:356] "Generic (PLEG): container finished" podID="b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" containerID="40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45" exitCode=0
Oct 30 19:22:59.047311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.047149    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"
Oct 30 19:22:59.047311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.047173    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq" event={"ID":"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908","Type":"ContainerDied","Data":"40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45"}
Oct 30 19:22:59.047311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.047219    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq" event={"ID":"b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908","Type":"ContainerDied","Data":"6ca54e208957d58175226ffbb194f7504189c02c77152f617201fa639fdc90de"}
Oct 30 19:22:59.047311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.047240    4267 scope.go:122] "RemoveContainer" containerID="40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45"
Oct 30 19:22:59.058732 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.058718    4267 scope.go:122] "RemoveContainer" containerID="40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45"
Oct 30 19:22:59.059012 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:22:59.058992    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45\": container with ID starting with 40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45 not found: ID does not exist" containerID="40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45"
Oct 30 19:22:59.059066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.059021    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45"} err="failed to get container status \"40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45\": rpc error: code = NotFound desc = could not find container \"40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45\": container with ID starting with 40445e5c32ca8e1569241084115fbbd052f4552f36449d182f1a9f3291c07c45 not found: ID does not exist"
Oct 30 19:22:59.211915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.211875    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"]
Oct 30 19:22:59.261603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:22:59.261567    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jdgbq"]
Oct 30 19:23:01.219877 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:01.219842    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" path="/var/lib/kubelet/pods/b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908/volumes"
Oct 30 19:23:08.794534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:08.794469    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd" podUID="b52af405-150a-4af8-9bde-a9afad214d47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:12.554452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:12.554417    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 19:23:12.657089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:12.656996    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b52af405-150a-4af8-9bde-a9afad214d47-kube-api-access-5f7h9\" (UniqueName: \"kubernetes.io/projected/b52af405-150a-4af8-9bde-a9afad214d47-kube-api-access-5f7h9\") pod \"b52af405-150a-4af8-9bde-a9afad214d47\" (UID: \"b52af405-150a-4af8-9bde-a9afad214d47\") "
Oct 30 19:23:12.657089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:12.657059    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b52af405-150a-4af8-9bde-a9afad214d47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b52af405-150a-4af8-9bde-a9afad214d47-plugins-config-volume\") pod \"b52af405-150a-4af8-9bde-a9afad214d47\" (UID: \"b52af405-150a-4af8-9bde-a9afad214d47\") "
Oct 30 19:23:12.657458 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:12.657427    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52af405-150a-4af8-9bde-a9afad214d47-plugins-config-volume" pod "b52af405-150a-4af8-9bde-a9afad214d47" (UID: "b52af405-150a-4af8-9bde-a9afad214d47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:12.663522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:12.659566    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52af405-150a-4af8-9bde-a9afad214d47-kube-api-access-5f7h9" pod "b52af405-150a-4af8-9bde-a9afad214d47" (UID: "b52af405-150a-4af8-9bde-a9afad214d47"). InnerVolumeSpecName "kube-api-access-5f7h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:12.758145 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:12.758100    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7h9\" (UniqueName: \"kubernetes.io/projected/b52af405-150a-4af8-9bde-a9afad214d47-kube-api-access-5f7h9\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:12.758145 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:12.758140    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b52af405-150a-4af8-9bde-a9afad214d47-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:13.132058 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:13.132018    4267 generic.go:356] "Generic (PLEG): container finished" podID="b52af405-150a-4af8-9bde-a9afad214d47" containerID="83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4" exitCode=0
Oct 30 19:23:13.132058 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:13.132067    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"
Oct 30 19:23:13.132345 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:13.132097    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd" event={"ID":"b52af405-150a-4af8-9bde-a9afad214d47","Type":"ContainerDied","Data":"83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4"}
Oct 30 19:23:13.132345 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:13.132143    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd" event={"ID":"b52af405-150a-4af8-9bde-a9afad214d47","Type":"ContainerDied","Data":"c5a94913ebc1f9db4265802237d4dd5e381b1166b5b5a0b8c9235295196074aa"}
Oct 30 19:23:13.132345 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:13.132168    4267 scope.go:122] "RemoveContainer" containerID="83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4"
Oct 30 19:23:13.141812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:13.141789    4267 scope.go:122] "RemoveContainer" containerID="83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4"
Oct 30 19:23:13.142116 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:23:13.142097    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4\": container with ID starting with 83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4 not found: ID does not exist" containerID="83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4"
Oct 30 19:23:13.142206 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:13.142128    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4"} err="failed to get container status \"83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4\": rpc error: code = NotFound desc = could not find container \"83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4\": container with ID starting with 83b64218f344a1b6af14e8e685bcef134f07d6a393a7ae23c6f1bea04dc1c6a4 not found: ID does not exist"
Oct 30 19:23:13.296517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:13.296459    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"]
Oct 30 19:23:13.346076 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:13.346038    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-66bdc0-5576bd8899-tbjqd"]
Oct 30 19:23:15.220336 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:15.220295    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b52af405-150a-4af8-9bde-a9afad214d47" path="/var/lib/kubelet/pods/b52af405-150a-4af8-9bde-a9afad214d47/volumes"
Oct 30 19:23:15.657238 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:15.657131    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"]
Oct 30 19:23:15.657436 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:15.657373    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy" containerID="cri-o://53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9" gracePeriod=30
Oct 30 19:23:20.158162 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:23:20.158118    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:23:20.908031 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:20.908004    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 19:23:21.040124 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040091    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-kube-api-access-96n2p\" (UniqueName: \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-kube-api-access-96n2p\") pod \"87fb6889-da6f-4d12-984f-4d0a6f217f05\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") "
Oct 30 19:23:21.040322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040144    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-credential-socket\") pod \"87fb6889-da6f-4d12-984f-4d0a6f217f05\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") "
Oct 30 19:23:21.040322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040212    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-certs\") pod \"87fb6889-da6f-4d12-984f-4d0a6f217f05\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") "
Oct 30 19:23:21.040322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040253    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-socket\") pod \"87fb6889-da6f-4d12-984f-4d0a6f217f05\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") "
Oct 30 19:23:21.040322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040298    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-data\") pod \"87fb6889-da6f-4d12-984f-4d0a6f217f05\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") "
Oct 30 19:23:21.040729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040326    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-podinfo\") pod \"87fb6889-da6f-4d12-984f-4d0a6f217f05\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") "
Oct 30 19:23:21.040729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040391    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-token\" (UniqueName: \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-token\") pod \"87fb6889-da6f-4d12-984f-4d0a6f217f05\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") "
Oct 30 19:23:21.040729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040422    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87fb6889-da6f-4d12-984f-4d0a6f217f05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87fb6889-da6f-4d12-984f-4d0a6f217f05-istiod-ca-cert\") pod \"87fb6889-da6f-4d12-984f-4d0a6f217f05\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") "
Oct 30 19:23:21.040729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040425    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-credential-socket" pod "87fb6889-da6f-4d12-984f-4d0a6f217f05" (UID: "87fb6889-da6f-4d12-984f-4d0a6f217f05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:21.040729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040602    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-certs" pod "87fb6889-da6f-4d12-984f-4d0a6f217f05" (UID: "87fb6889-da6f-4d12-984f-4d0a6f217f05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:21.040729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040616    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-data" pod "87fb6889-da6f-4d12-984f-4d0a6f217f05" (UID: "87fb6889-da6f-4d12-984f-4d0a6f217f05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:21.040729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.040631    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-envoy\") pod \"87fb6889-da6f-4d12-984f-4d0a6f217f05\" (UID: \"87fb6889-da6f-4d12-984f-4d0a6f217f05\") "
Oct 30 19:23:21.041785 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.041483    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-socket" pod "87fb6889-da6f-4d12-984f-4d0a6f217f05" (UID: "87fb6889-da6f-4d12-984f-4d0a6f217f05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:21.041785 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.041704    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87fb6889-da6f-4d12-984f-4d0a6f217f05-istiod-ca-cert" pod "87fb6889-da6f-4d12-984f-4d0a6f217f05" (UID: "87fb6889-da6f-4d12-984f-4d0a6f217f05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:21.042434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.042018    4267 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-credential-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:21.042434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.042050    4267 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:21.042434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.042070    4267 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-workload-socket\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:21.042434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.042131    4267 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-data\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:21.052509 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.052281    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-kube-api-access-96n2p" pod "87fb6889-da6f-4d12-984f-4d0a6f217f05" (UID: "87fb6889-da6f-4d12-984f-4d0a6f217f05"). InnerVolumeSpecName "kube-api-access-96n2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:21.052509 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.052345    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-podinfo" pod "87fb6889-da6f-4d12-984f-4d0a6f217f05" (UID: "87fb6889-da6f-4d12-984f-4d0a6f217f05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:23:21.052696 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.052557    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-token" pod "87fb6889-da6f-4d12-984f-4d0a6f217f05" (UID: "87fb6889-da6f-4d12-984f-4d0a6f217f05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:21.053211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.053184    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-envoy" pod "87fb6889-da6f-4d12-984f-4d0a6f217f05" (UID: "87fb6889-da6f-4d12-984f-4d0a6f217f05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:21.143468 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.143430    4267 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-podinfo\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:21.143468 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.143466    4267 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-token\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:21.143683 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.143478    4267 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87fb6889-da6f-4d12-984f-4d0a6f217f05-istiod-ca-cert\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:21.143683 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.143514    4267 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87fb6889-da6f-4d12-984f-4d0a6f217f05-istio-envoy\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:21.143683 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.143526    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96n2p\" (UniqueName: \"kubernetes.io/projected/87fb6889-da6f-4d12-984f-4d0a6f217f05-kube-api-access-96n2p\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:21.184914 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.184880    4267 generic.go:356] "Generic (PLEG): container finished" podID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerID="53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9" exitCode=0
Oct 30 19:23:21.185356 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.184948    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"
Oct 30 19:23:21.185356 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.184957    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" event={"ID":"87fb6889-da6f-4d12-984f-4d0a6f217f05","Type":"ContainerDied","Data":"53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9"}
Oct 30 19:23:21.185356 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.185048    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz" event={"ID":"87fb6889-da6f-4d12-984f-4d0a6f217f05","Type":"ContainerDied","Data":"2a5bb7bbd0a6059f2b9433ded8ce0fc837896b5e90755953b59071586d1b7549"}
Oct 30 19:23:21.185356 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.185079    4267 scope.go:122] "RemoveContainer" containerID="53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9"
Oct 30 19:23:21.196480 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.196459    4267 scope.go:122] "RemoveContainer" containerID="3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8"
Oct 30 19:23:21.204748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.204726    4267 scope.go:122] "RemoveContainer" containerID="53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9"
Oct 30 19:23:21.204991 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:23:21.204969    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9\": container with ID starting with 53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9 not found: ID does not exist" containerID="53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9"
Oct 30 19:23:21.205076 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.205000    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9"} err="failed to get container status \"53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9\": rpc error: code = NotFound desc = could not find container \"53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9\": container with ID starting with 53322172584afc18583a985ec477de9d084d2fb739748e88b1760ad10f3386e9 not found: ID does not exist"
Oct 30 19:23:21.205076 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.205019    4267 scope.go:122] "RemoveContainer" containerID="3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8"
Oct 30 19:23:21.205249 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:23:21.205227    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8\": container with ID starting with 3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8 not found: ID does not exist" containerID="3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8"
Oct 30 19:23:21.205350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.205254    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8"} err="failed to get container status \"3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8\": rpc error: code = NotFound desc = could not find container \"3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8\": container with ID starting with 3a494925fee1e17053151e45806032467f9f7c021c20abab4809fb8fc00714b8 not found: ID does not exist"
Oct 30 19:23:21.350584 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.350488    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"]
Oct 30 19:23:21.354052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:21.354025    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66bdc0-openshift-default-7db494c579-bjdfz"]
Oct 30 19:23:23.066875 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:23.065961    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"]
Oct 30 19:23:23.114776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:23.114730    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a2f0cc-6d6g9"]
Oct 30 19:23:23.222447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:23.222411    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" path="/var/lib/kubelet/pods/87fb6889-da6f-4d12-984f-4d0a6f217f05/volumes"
Oct 30 19:23:23.223083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:23.223059    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d641d9-e712-4a48-ad72-b0c2443b309d" path="/var/lib/kubelet/pods/e5d641d9-e712-4a48-ad72-b0c2443b309d/volumes"
Oct 30 19:23:31.770416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:31.770373    4267 scope.go:122] "RemoveContainer" containerID="4a2b9468728d03a900920cf1b7c0d20e6d3324f1679d33b9839ca3d8653a5ff6"
Oct 30 19:23:31.779250 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:31.779227    4267 scope.go:122] "RemoveContainer" containerID="81dbc80e7b649abdef77f81fcb8c8bdf321a7e4eca4762d983950787b5bf9205"
Oct 30 19:23:34.047405 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:34.047364    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"]
Oct 30 19:23:34.096929 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:34.096890    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8d3955-wr2fq"]
Oct 30 19:23:35.219112 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:35.219076    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d304bea0-f8b8-43b5-a243-c49592c152ac" path="/var/lib/kubelet/pods/d304bea0-f8b8-43b5-a243-c49592c152ac/volumes"
Oct 30 19:23:47.031211 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031176    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"]
Oct 30 19:23:47.031657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031543    4267 cpu_manager.go:414] "containerMap: removing container" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy"
Oct 30 19:23:47.031657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031554    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy"
Oct 30 19:23:47.031657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031568    4267 cpu_manager.go:414] "containerMap: removing container" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main"
Oct 30 19:23:47.031657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031573    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main"
Oct 30 19:23:47.031657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031586    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b52af405-150a-4af8-9bde-a9afad214d47" containerName="payload-pre-processing"
Oct 30 19:23:47.031657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031592    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b52af405-150a-4af8-9bde-a9afad214d47" containerName="payload-pre-processing"
Oct 30 19:23:47.031657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031615    4267 cpu_manager.go:414] "containerMap: removing container" podUID="8a1b4aa0-90f3-4b8b-a627-c0619805940f" containerName="revoke-keys"
Oct 30 19:23:47.031657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031620    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1b4aa0-90f3-4b8b-a627-c0619805940f" containerName="revoke-keys"
Oct 30 19:23:47.031657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031626    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" containerName="discovery"
Oct 30 19:23:47.031657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031631    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" containerName="discovery"
Oct 30 19:23:47.031993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031681    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy"
Oct 30 19:23:47.031993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031690    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy"
Oct 30 19:23:47.031993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031697    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="a511f699-9b06-4893-a9f7-acb661ceb6c0" containerName="main"
Oct 30 19:23:47.031993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031706    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b52af405-150a-4af8-9bde-a9afad214d47" containerName="payload-pre-processing"
Oct 30 19:23:47.031993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031716    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1b4aa0-90f3-4b8b-a627-c0619805940f" containerName="revoke-keys"
Oct 30 19:23:47.031993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.031722    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ea6e52-1ff7-4ca4-9c9c-6081c0f1a908" containerName="discovery"
Oct 30 19:23:47.036237 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.036219    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.139125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.139088    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:47.139125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.139092    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:23:47.139400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.139152    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:23:47.139400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.139194    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:23:47.141813 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.141787    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"]
Oct 30 19:23:47.144530 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.144484    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4p6v\" (UniqueName: \"kubernetes.io/projected/974108d4-dbc0-49f2-b013-80e7e8603a8f-kube-api-access-s4p6v\") pod \"maas-api-revoke-sub-85237e45-26tvl\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.144610 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.144547    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-26tvl\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.144681 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.144656    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-26tvl\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.245959 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.245922    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4p6v\" (UniqueName: \"kubernetes.io/projected/974108d4-dbc0-49f2-b013-80e7e8603a8f-kube-api-access-s4p6v\") pod \"maas-api-revoke-sub-85237e45-26tvl\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.246144 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.245967    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-26tvl\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.246144 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.246006    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-26tvl\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.246679 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.246655    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-26tvl\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.248385 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.248362    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-26tvl\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.303763 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.303685    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4p6v\" (UniqueName: \"kubernetes.io/projected/974108d4-dbc0-49f2-b013-80e7e8603a8f-kube-api-access-s4p6v\") pod \"maas-api-revoke-sub-85237e45-26tvl\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.349698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.349668    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:47.732469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:47.732440    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"]
Oct 30 19:23:47.733326 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:23:47.733293    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974108d4_dbc0_49f2_b013_80e7e8603a8f.slice/crio-d83de240fcd82a828d57140759997b0b56986261b303c73a33a4fe2ea0943983 WatchSource:0}: Error finding container d83de240fcd82a828d57140759997b0b56986261b303c73a33a4fe2ea0943983: Status 404 returned error can't find the container with id d83de240fcd82a828d57140759997b0b56986261b303c73a33a4fe2ea0943983
Oct 30 19:23:48.384587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:48.384473    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl" event={"ID":"974108d4-dbc0-49f2-b013-80e7e8603a8f","Type":"ContainerStarted","Data":"27168df8906d3b8741c368c8c065fb7f6114ef89a15a9018ce7c426373b602ca"}
Oct 30 19:23:48.384587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:48.384556    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl" event={"ID":"974108d4-dbc0-49f2-b013-80e7e8603a8f","Type":"ContainerStarted","Data":"d83de240fcd82a828d57140759997b0b56986261b303c73a33a4fe2ea0943983"}
Oct 30 19:23:48.498729 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:48.498673    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl" podStartSLOduration=2.49865674 podStartE2EDuration="2.49865674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:48.498118599 +0000 UTC m=+2719.386035688" watchObservedRunningTime="2026-10-30 19:23:48.49865674 +0000 UTC m=+2719.386573828"
Oct 30 19:23:49.391617 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:49.391584    4267 generic.go:356] "Generic (PLEG): container finished" podID="974108d4-dbc0-49f2-b013-80e7e8603a8f" containerID="27168df8906d3b8741c368c8c065fb7f6114ef89a15a9018ce7c426373b602ca" exitCode=0
Oct 30 19:23:49.391999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:49.391667    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl" event={"ID":"974108d4-dbc0-49f2-b013-80e7e8603a8f","Type":"ContainerDied","Data":"27168df8906d3b8741c368c8c065fb7f6114ef89a15a9018ce7c426373b602ca"}
Oct 30 19:23:50.161154 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:23:50.161110    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:23:50.533167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:50.533133    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:50.677993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:50.677954    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974108d4-dbc0-49f2-b013-80e7e8603a8f-kube-api-access-s4p6v\" (UniqueName: \"kubernetes.io/projected/974108d4-dbc0-49f2-b013-80e7e8603a8f-kube-api-access-s4p6v\") pod \"974108d4-dbc0-49f2-b013-80e7e8603a8f\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") "
Oct 30 19:23:50.678217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:50.678054    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-openshift-ca\") pod \"974108d4-dbc0-49f2-b013-80e7e8603a8f\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") "
Oct 30 19:23:50.678217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:50.678103    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-xks-ca\") pod \"974108d4-dbc0-49f2-b013-80e7e8603a8f\" (UID: \"974108d4-dbc0-49f2-b013-80e7e8603a8f\") "
Oct 30 19:23:50.678508 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:50.678469    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-openshift-ca" pod "974108d4-dbc0-49f2-b013-80e7e8603a8f" (UID: "974108d4-dbc0-49f2-b013-80e7e8603a8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:50.680164 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:50.680138    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974108d4-dbc0-49f2-b013-80e7e8603a8f-kube-api-access-s4p6v" pod "974108d4-dbc0-49f2-b013-80e7e8603a8f" (UID: "974108d4-dbc0-49f2-b013-80e7e8603a8f"). InnerVolumeSpecName "kube-api-access-s4p6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:50.680264 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:50.680215    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-xks-ca" pod "974108d4-dbc0-49f2-b013-80e7e8603a8f" (UID: "974108d4-dbc0-49f2-b013-80e7e8603a8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:50.779999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:50.779912    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:50.779999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:50.779946    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/974108d4-dbc0-49f2-b013-80e7e8603a8f-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:50.779999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:50.779957    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4p6v\" (UniqueName: \"kubernetes.io/projected/974108d4-dbc0-49f2-b013-80e7e8603a8f-kube-api-access-s4p6v\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:23:51.406005 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:51.405976    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"
Oct 30 19:23:51.406005 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:51.405981    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl" event={"ID":"974108d4-dbc0-49f2-b013-80e7e8603a8f","Type":"ContainerDied","Data":"d83de240fcd82a828d57140759997b0b56986261b303c73a33a4fe2ea0943983"}
Oct 30 19:23:51.406283 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:51.406025    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d83de240fcd82a828d57140759997b0b56986261b303c73a33a4fe2ea0943983"
Oct 30 19:23:57.049946 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:57.049908    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"]
Oct 30 19:23:57.099662 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:57.099623    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686e2376-627dp"]
Oct 30 19:23:57.219565 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:23:57.219527    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4253ad0-ff62-4ac3-9b21-68644674c4f9" path="/var/lib/kubelet/pods/c4253ad0-ff62-4ac3-9b21-68644674c4f9/volumes"
Oct 30 19:24:20.158241 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:24:20.158138    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:24:31.823741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:31.823704    4267 scope.go:122] "RemoveContainer" containerID="663c8f4474624769f7ea38f361efbdac1e195f2de7edc1e6cc550bb692cafeb2"
Oct 30 19:24:31.832879 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:31.832857    4267 scope.go:122] "RemoveContainer" containerID="39ef61a91a34457d5cac782936805f14d956b34c3777b3eee6898365110400aa"
Oct 30 19:24:31.842034 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:31.842009    4267 scope.go:122] "RemoveContainer" containerID="c2bd0abde857e5e6d724d2159f35a4d1bc6da64116b923194d2d47929356239a"
Oct 30 19:24:31.850629 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:31.850604    4267 scope.go:122] "RemoveContainer" containerID="76ac98888cec4d9f91863b43276dc04f9ef029cc64e817577abe1dcd5fd79047"
Oct 30 19:24:31.858649 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:31.858631    4267 scope.go:122] "RemoveContainer" containerID="c8d7e3d899baeb557a57028d9f05a730fef0516d1ad0a4f1fb2394bb0b63d007"
Oct 30 19:24:46.476957 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.476909    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"]
Oct 30 19:24:46.477537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.477517    4267 cpu_manager.go:414] "containerMap: removing container" podUID="974108d4-dbc0-49f2-b013-80e7e8603a8f" containerName="revoke-keys"
Oct 30 19:24:46.477626 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.477538    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974108d4-dbc0-49f2-b013-80e7e8603a8f" containerName="revoke-keys"
Oct 30 19:24:46.477626 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.477557    4267 cpu_manager.go:414] "containerMap: removing container" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy"
Oct 30 19:24:46.477626 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.477565    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87fb6889-da6f-4d12-984f-4d0a6f217f05" containerName="istio-proxy"
Oct 30 19:24:46.477799 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.477683    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="974108d4-dbc0-49f2-b013-80e7e8603a8f" containerName="revoke-keys"
Oct 30 19:24:46.482762 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.482738    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:46.536054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.536014    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:24:46.545559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.545522    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fmr7\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:46.545726 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.545639    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fmr7\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:46.545726 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.545705    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgm55\" (UniqueName: \"kubernetes.io/projected/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-kube-api-access-xgm55\") pod \"maas-api-revoke-sub-c1e271a9-7fmr7\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:46.585472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.585436    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:24:46.585472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.585452    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:46.585472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.585468    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:24:46.588207 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.588179    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"]
Oct 30 19:24:46.646711 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.646672    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fmr7\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:46.646915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.646720    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgm55\" (UniqueName: \"kubernetes.io/projected/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-kube-api-access-xgm55\") pod \"maas-api-revoke-sub-c1e271a9-7fmr7\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:46.646915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.646811    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fmr7\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:46.647341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.647319    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fmr7\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:46.649161 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.649136    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fmr7\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:46.704226 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.704190    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgm55\" (UniqueName: \"kubernetes.io/projected/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-kube-api-access-xgm55\") pod \"maas-api-revoke-sub-c1e271a9-7fmr7\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:46.795461 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:46.795369    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:47.168904 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:47.168877    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"]
Oct 30 19:24:47.170445 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:24:47.170413    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9b3fa9_6c20_4a41_a2a0_37544c5f944c.slice/crio-c1a1dbeb86d06790d288f0e30b5a2530bb0aacc8df34907ab3216b91a0b85f16 WatchSource:0}: Error finding container c1a1dbeb86d06790d288f0e30b5a2530bb0aacc8df34907ab3216b91a0b85f16: Status 404 returned error can't find the container with id c1a1dbeb86d06790d288f0e30b5a2530bb0aacc8df34907ab3216b91a0b85f16
Oct 30 19:24:47.730596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:47.730554    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7" event={"ID":"af9b3fa9-6c20-4a41-a2a0-37544c5f944c","Type":"ContainerStarted","Data":"a0af4543acca5cb92e3333758f09107f565af8d452411a0ff73eb8a16ff0d769"}
Oct 30 19:24:47.731013 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:47.730616    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7" event={"ID":"af9b3fa9-6c20-4a41-a2a0-37544c5f944c","Type":"ContainerStarted","Data":"c1a1dbeb86d06790d288f0e30b5a2530bb0aacc8df34907ab3216b91a0b85f16"}
Oct 30 19:24:47.846760 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:47.846696    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7" podStartSLOduration=1.8466804209999998 podStartE2EDuration="1.846680421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:47.844176517 +0000 UTC m=+2778.732093606" watchObservedRunningTime="2026-10-30 19:24:47.846680421 +0000 UTC m=+2778.734597509"
Oct 30 19:24:48.737906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:48.737869    4267 generic.go:356] "Generic (PLEG): container finished" podID="af9b3fa9-6c20-4a41-a2a0-37544c5f944c" containerID="a0af4543acca5cb92e3333758f09107f565af8d452411a0ff73eb8a16ff0d769" exitCode=0
Oct 30 19:24:48.737906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:48.737919    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7" event={"ID":"af9b3fa9-6c20-4a41-a2a0-37544c5f944c","Type":"ContainerDied","Data":"a0af4543acca5cb92e3333758f09107f565af8d452411a0ff73eb8a16ff0d769"}
Oct 30 19:24:49.877394 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:49.877370    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:49.979204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:49.979171    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-xks-ca\") pod \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") "
Oct 30 19:24:49.979417 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:49.979258    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-kube-api-access-xgm55\" (UniqueName: \"kubernetes.io/projected/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-kube-api-access-xgm55\") pod \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") "
Oct 30 19:24:49.979417 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:49.979319    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-openshift-ca\") pod \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\" (UID: \"af9b3fa9-6c20-4a41-a2a0-37544c5f944c\") "
Oct 30 19:24:49.979739 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:49.979719    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-openshift-ca" pod "af9b3fa9-6c20-4a41-a2a0-37544c5f944c" (UID: "af9b3fa9-6c20-4a41-a2a0-37544c5f944c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:49.981365 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:49.981336    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-xks-ca" pod "af9b3fa9-6c20-4a41-a2a0-37544c5f944c" (UID: "af9b3fa9-6c20-4a41-a2a0-37544c5f944c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:49.981481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:49.981421    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-kube-api-access-xgm55" pod "af9b3fa9-6c20-4a41-a2a0-37544c5f944c" (UID: "af9b3fa9-6c20-4a41-a2a0-37544c5f944c"). InnerVolumeSpecName "kube-api-access-xgm55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:50.047685 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:50.047518    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"]
Oct 30 19:24:50.050592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:50.050525    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32f8257b-wxf8s"]
Oct 30 19:24:50.080750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:50.080715    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:50.080750 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:50.080745    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:50.080935 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:50.080757    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgm55\" (UniqueName: \"kubernetes.io/projected/af9b3fa9-6c20-4a41-a2a0-37544c5f944c-kube-api-access-xgm55\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:24:50.158157 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:24:50.158115    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:24:50.750463 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:50.750432    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"
Oct 30 19:24:50.750647 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:50.750474    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7" event={"ID":"af9b3fa9-6c20-4a41-a2a0-37544c5f944c","Type":"ContainerDied","Data":"c1a1dbeb86d06790d288f0e30b5a2530bb0aacc8df34907ab3216b91a0b85f16"}
Oct 30 19:24:50.750647 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:50.750542    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1a1dbeb86d06790d288f0e30b5a2530bb0aacc8df34907ab3216b91a0b85f16"
Oct 30 19:24:51.045963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:51.045878    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"]
Oct 30 19:24:51.054907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:51.050409    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a0ae7d-4ppnc"]
Oct 30 19:24:51.219808 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:51.219776    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c1a2ea-9388-4576-8982-4eabbce9c52e" path="/var/lib/kubelet/pods/62c1a2ea-9388-4576-8982-4eabbce9c52e/volumes"
Oct 30 19:24:51.220204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:24:51.220192    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a51ae4-9150-4918-bc06-2876614af3a1" path="/var/lib/kubelet/pods/b9a51ae4-9150-4918-bc06-2876614af3a1/volumes"
Oct 30 19:25:14.647297 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.647260    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"]
Oct 30 19:25:14.647960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.647861    4267 cpu_manager.go:414] "containerMap: removing container" podUID="af9b3fa9-6c20-4a41-a2a0-37544c5f944c" containerName="revoke-keys"
Oct 30 19:25:14.647960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.647884    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9b3fa9-6c20-4a41-a2a0-37544c5f944c" containerName="revoke-keys"
Oct 30 19:25:14.648086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.647971    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9b3fa9-6c20-4a41-a2a0-37544c5f944c" containerName="revoke-keys"
Oct 30 19:25:14.650985 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.650965    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:14.755074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.755039    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:25:14.755074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.755065    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:14.755353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.755264    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:25:14.755353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.755305    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:25:14.756314 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.756294    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"]
Oct 30 19:25:14.766763 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.766740    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rzgk\" (UniqueName: \"kubernetes.io/projected/4f92b12a-193f-44bb-afc7-1f9501266f1c-kube-api-access-8rzgk\") pod \"maas-api-revoke-sub-2e1f5b5f-286lk\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:14.766877 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.766858    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-286lk\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:14.766935 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.766920    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-286lk\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:14.867894 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.867860    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rzgk\" (UniqueName: \"kubernetes.io/projected/4f92b12a-193f-44bb-afc7-1f9501266f1c-kube-api-access-8rzgk\") pod \"maas-api-revoke-sub-2e1f5b5f-286lk\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:14.868095 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.868022    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-286lk\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:14.868095 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.868057    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-286lk\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:14.868734 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.868712    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-286lk\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:14.870357 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.870331    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-286lk\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:14.925732 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.925665    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rzgk\" (UniqueName: \"kubernetes.io/projected/4f92b12a-193f-44bb-afc7-1f9501266f1c-kube-api-access-8rzgk\") pod \"maas-api-revoke-sub-2e1f5b5f-286lk\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:14.964425 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:14.964391    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:15.340456 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:15.340425    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"]
Oct 30 19:25:15.342003 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:25:15.341974    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f92b12a_193f_44bb_afc7_1f9501266f1c.slice/crio-408eebe3e05c9954ea9c72efa52293c6c2a7e6bc01dcd813105cfbf3a5823659 WatchSource:0}: Error finding container 408eebe3e05c9954ea9c72efa52293c6c2a7e6bc01dcd813105cfbf3a5823659: Status 404 returned error can't find the container with id 408eebe3e05c9954ea9c72efa52293c6c2a7e6bc01dcd813105cfbf3a5823659
Oct 30 19:25:15.926999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:15.926908    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk" event={"ID":"4f92b12a-193f-44bb-afc7-1f9501266f1c","Type":"ContainerStarted","Data":"828b00de9c5517f714f2ffebcfd79a9ea9f5b460426deb6c5410f96579bb0fe4"}
Oct 30 19:25:15.926999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:15.926970    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk" event={"ID":"4f92b12a-193f-44bb-afc7-1f9501266f1c","Type":"ContainerStarted","Data":"408eebe3e05c9954ea9c72efa52293c6c2a7e6bc01dcd813105cfbf3a5823659"}
Oct 30 19:25:16.042188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:16.042132    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk" podStartSLOduration=2.042118531 podStartE2EDuration="2.042118531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:16.042048663 +0000 UTC m=+2806.929965751" watchObservedRunningTime="2026-10-30 19:25:16.042118531 +0000 UTC m=+2806.930035620"
Oct 30 19:25:16.933753 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:16.933721    4267 generic.go:356] "Generic (PLEG): container finished" podID="4f92b12a-193f-44bb-afc7-1f9501266f1c" containerID="828b00de9c5517f714f2ffebcfd79a9ea9f5b460426deb6c5410f96579bb0fe4" exitCode=0
Oct 30 19:25:16.934176 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:16.933819    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk" event={"ID":"4f92b12a-193f-44bb-afc7-1f9501266f1c","Type":"ContainerDied","Data":"828b00de9c5517f714f2ffebcfd79a9ea9f5b460426deb6c5410f96579bb0fe4"}
Oct 30 19:25:18.077156 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.077130    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:18.200703 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.200624    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-openshift-ca\") pod \"4f92b12a-193f-44bb-afc7-1f9501266f1c\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") "
Oct 30 19:25:18.200861 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.200746    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-xks-ca\") pod \"4f92b12a-193f-44bb-afc7-1f9501266f1c\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") "
Oct 30 19:25:18.200861 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.200793    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f92b12a-193f-44bb-afc7-1f9501266f1c-kube-api-access-8rzgk\" (UniqueName: \"kubernetes.io/projected/4f92b12a-193f-44bb-afc7-1f9501266f1c-kube-api-access-8rzgk\") pod \"4f92b12a-193f-44bb-afc7-1f9501266f1c\" (UID: \"4f92b12a-193f-44bb-afc7-1f9501266f1c\") "
Oct 30 19:25:18.201080 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.201049    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-openshift-ca" pod "4f92b12a-193f-44bb-afc7-1f9501266f1c" (UID: "4f92b12a-193f-44bb-afc7-1f9501266f1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:18.202845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.202822    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-xks-ca" pod "4f92b12a-193f-44bb-afc7-1f9501266f1c" (UID: "4f92b12a-193f-44bb-afc7-1f9501266f1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:18.202954 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.202939    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f92b12a-193f-44bb-afc7-1f9501266f1c-kube-api-access-8rzgk" pod "4f92b12a-193f-44bb-afc7-1f9501266f1c" (UID: "4f92b12a-193f-44bb-afc7-1f9501266f1c"). InnerVolumeSpecName "kube-api-access-8rzgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:18.301868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.301826    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.301868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.301863    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rzgk\" (UniqueName: \"kubernetes.io/projected/4f92b12a-193f-44bb-afc7-1f9501266f1c-kube-api-access-8rzgk\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.301868 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.301875    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f92b12a-193f-44bb-afc7-1f9501266f1c-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:18.945902 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.945868    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"
Oct 30 19:25:18.946091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.945900    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk" event={"ID":"4f92b12a-193f-44bb-afc7-1f9501266f1c","Type":"ContainerDied","Data":"408eebe3e05c9954ea9c72efa52293c6c2a7e6bc01dcd813105cfbf3a5823659"}
Oct 30 19:25:18.946091 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:18.945948    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="408eebe3e05c9954ea9c72efa52293c6c2a7e6bc01dcd813105cfbf3a5823659"
Oct 30 19:25:20.158005 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:25:20.157968    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:25:22.047984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:22.047947    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"]
Oct 30 19:25:22.097430 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:22.097396    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"]
Oct 30 19:25:22.099831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:22.099800    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5babae40-nqjmg"]
Oct 30 19:25:22.101651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:22.101630    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81255a89-fdlp5"]
Oct 30 19:25:23.218882 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:23.218849    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227bb3dc-ffe9-4128-8ca3-ff4955cf8327" path="/var/lib/kubelet/pods/227bb3dc-ffe9-4128-8ca3-ff4955cf8327/volumes"
Oct 30 19:25:23.219279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:23.219271    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ffcbb7-526d-4a6c-842f-d7112cd0d942" path="/var/lib/kubelet/pods/45ffcbb7-526d-4a6c-842f-d7112cd0d942/volumes"
Oct 30 19:25:26.958853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:26.958812    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"]
Oct 30 19:25:26.959266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:26.959197    4267 cpu_manager.go:414] "containerMap: removing container" podUID="4f92b12a-193f-44bb-afc7-1f9501266f1c" containerName="revoke-keys"
Oct 30 19:25:26.959266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:26.959208    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f92b12a-193f-44bb-afc7-1f9501266f1c" containerName="revoke-keys"
Oct 30 19:25:26.959266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:26.959263    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f92b12a-193f-44bb-afc7-1f9501266f1c" containerName="revoke-keys"
Oct 30 19:25:26.963574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:26.963555    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:26.983706 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:26.983681    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qw9q9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:26.983853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:26.983724    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qw9q9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:26.983917 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:26.983864    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4k7b\" (UniqueName: \"kubernetes.io/projected/6f84332a-be22-4aaf-8506-8780fe73d1d9-kube-api-access-s4k7b\") pod \"maas-api-revoke-sub-ec843018-qw9q9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:27.065641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.065608    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:27.065641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.065614    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:25:27.065849 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.065677    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:25:27.065849 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.065716    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:25:27.068306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.068282    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"]
Oct 30 19:25:27.084511 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.084462    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qw9q9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:27.084631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.084573    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4k7b\" (UniqueName: \"kubernetes.io/projected/6f84332a-be22-4aaf-8506-8780fe73d1d9-kube-api-access-s4k7b\") pod \"maas-api-revoke-sub-ec843018-qw9q9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:27.084674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.084638    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qw9q9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:27.085220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.085199    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-qw9q9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:27.087023 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.087003    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-qw9q9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:27.141422 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.141391    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4k7b\" (UniqueName: \"kubernetes.io/projected/6f84332a-be22-4aaf-8506-8780fe73d1d9-kube-api-access-s4k7b\") pod \"maas-api-revoke-sub-ec843018-qw9q9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:27.276912 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.276833    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:27.648636 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.648609    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"]
Oct 30 19:25:27.650069 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:25:27.650035    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f84332a_be22_4aaf_8506_8780fe73d1d9.slice/crio-b6a14bdde86a757ef8faaaef73ca9970f2cdc21bb0fbcd4d98b6f789487f3e99 WatchSource:0}: Error finding container b6a14bdde86a757ef8faaaef73ca9970f2cdc21bb0fbcd4d98b6f789487f3e99: Status 404 returned error can't find the container with id b6a14bdde86a757ef8faaaef73ca9970f2cdc21bb0fbcd4d98b6f789487f3e99
Oct 30 19:25:27.995324 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.995283    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9" event={"ID":"6f84332a-be22-4aaf-8506-8780fe73d1d9","Type":"ContainerStarted","Data":"c866e72959eeb86c34cde89e3ba4042b2c959ebcc059d6d959506b7dac186bd6"}
Oct 30 19:25:27.995768 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:27.995350    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9" event={"ID":"6f84332a-be22-4aaf-8506-8780fe73d1d9","Type":"ContainerStarted","Data":"b6a14bdde86a757ef8faaaef73ca9970f2cdc21bb0fbcd4d98b6f789487f3e99"}
Oct 30 19:25:28.124361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:28.124310    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9" podStartSLOduration=2.12429482 podStartE2EDuration="2.12429482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:28.122069634 +0000 UTC m=+2819.009986714" watchObservedRunningTime="2026-10-30 19:25:28.12429482 +0000 UTC m=+2819.012211909"
Oct 30 19:25:29.002836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:29.002800    4267 generic.go:356] "Generic (PLEG): container finished" podID="6f84332a-be22-4aaf-8506-8780fe73d1d9" containerID="c866e72959eeb86c34cde89e3ba4042b2c959ebcc059d6d959506b7dac186bd6" exitCode=0
Oct 30 19:25:29.003254 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:29.002887    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9" event={"ID":"6f84332a-be22-4aaf-8506-8780fe73d1d9","Type":"ContainerDied","Data":"c866e72959eeb86c34cde89e3ba4042b2c959ebcc059d6d959506b7dac186bd6"}
Oct 30 19:25:30.214020 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:30.213983    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:30.314008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:30.313884    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-openshift-ca\") pod \"6f84332a-be22-4aaf-8506-8780fe73d1d9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") "
Oct 30 19:25:30.314008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:30.313962    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f84332a-be22-4aaf-8506-8780fe73d1d9-kube-api-access-s4k7b\" (UniqueName: \"kubernetes.io/projected/6f84332a-be22-4aaf-8506-8780fe73d1d9-kube-api-access-s4k7b\") pod \"6f84332a-be22-4aaf-8506-8780fe73d1d9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") "
Oct 30 19:25:30.314298 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:30.314026    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-xks-ca\") pod \"6f84332a-be22-4aaf-8506-8780fe73d1d9\" (UID: \"6f84332a-be22-4aaf-8506-8780fe73d1d9\") "
Oct 30 19:25:30.314761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:30.314723    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-openshift-ca" pod "6f84332a-be22-4aaf-8506-8780fe73d1d9" (UID: "6f84332a-be22-4aaf-8506-8780fe73d1d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:30.317018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:30.316987    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-xks-ca" pod "6f84332a-be22-4aaf-8506-8780fe73d1d9" (UID: "6f84332a-be22-4aaf-8506-8780fe73d1d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:30.317844 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:30.317801    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f84332a-be22-4aaf-8506-8780fe73d1d9-kube-api-access-s4k7b" pod "6f84332a-be22-4aaf-8506-8780fe73d1d9" (UID: "6f84332a-be22-4aaf-8506-8780fe73d1d9"). InnerVolumeSpecName "kube-api-access-s4k7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:30.419526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:30.417078    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4k7b\" (UniqueName: \"kubernetes.io/projected/6f84332a-be22-4aaf-8506-8780fe73d1d9-kube-api-access-s4k7b\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:30.419526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:30.417121    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:30.419526 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:30.417138    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f84332a-be22-4aaf-8506-8780fe73d1d9-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:25:31.043335 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:31.043291    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9" event={"ID":"6f84332a-be22-4aaf-8506-8780fe73d1d9","Type":"ContainerDied","Data":"b6a14bdde86a757ef8faaaef73ca9970f2cdc21bb0fbcd4d98b6f789487f3e99"}
Oct 30 19:25:31.047512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:31.043650    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6a14bdde86a757ef8faaaef73ca9970f2cdc21bb0fbcd4d98b6f789487f3e99"
Oct 30 19:25:31.047512 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:31.043751    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"
Oct 30 19:25:31.905521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:31.898369    4267 scope.go:122] "RemoveContainer" containerID="e0eb9abb6533d1ca4590063683b09c24d14ffa7c1801c8de6f0cd566207a8b90"
Oct 30 19:25:31.958961 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:31.958923    4267 scope.go:122] "RemoveContainer" containerID="2870432d14c5e8e9d4d4efaf24af1ccab50b3de821787262f340f4118bb6d8c9"
Oct 30 19:25:32.008838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:32.008811    4267 scope.go:122] "RemoveContainer" containerID="e9068893019945d137c47c50d0684d6d976f33767d8af23d7305a01611966aa6"
Oct 30 19:25:48.046851 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:48.046767    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"]
Oct 30 19:25:48.096525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:48.096459    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c36105a9-6szp9"]
Oct 30 19:25:49.220400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:25:49.220364    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d78c2e2-6d38-4a0e-988c-1276f4ebd544" path="/var/lib/kubelet/pods/4d78c2e2-6d38-4a0e-988c-1276f4ebd544/volumes"
Oct 30 19:25:50.158224 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:25:50.158182    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:26:02.253357 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.253310    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"]
Oct 30 19:26:02.253997 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.253858    4267 cpu_manager.go:414] "containerMap: removing container" podUID="6f84332a-be22-4aaf-8506-8780fe73d1d9" containerName="revoke-keys"
Oct 30 19:26:02.253997 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.253881    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f84332a-be22-4aaf-8506-8780fe73d1d9" containerName="revoke-keys"
Oct 30 19:26:02.253997 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.253988    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f84332a-be22-4aaf-8506-8780fe73d1d9" containerName="revoke-keys"
Oct 30 19:26:02.257018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.256996    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.306931 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.306895    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxt4h\" (UniqueName: \"kubernetes.io/projected/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-kube-api-access-xxt4h\") pod \"maas-api-revoke-sub-046e1343-dhgj9\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.307129 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.307040    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dhgj9\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.307129 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.307083    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dhgj9\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.311476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.311441    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:26:02.361007 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.360961    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:26:02.361222 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.361065    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:02.361222 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.361087    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:26:02.363734 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.363708    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"]
Oct 30 19:26:02.408328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.408286    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dhgj9\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.408328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.408328    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dhgj9\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.408610 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.408388    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxt4h\" (UniqueName: \"kubernetes.io/projected/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-kube-api-access-xxt4h\") pod \"maas-api-revoke-sub-046e1343-dhgj9\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.409078 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.409056    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dhgj9\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.410901 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.410878    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dhgj9\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.466745 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.466711    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxt4h\" (UniqueName: \"kubernetes.io/projected/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-kube-api-access-xxt4h\") pod \"maas-api-revoke-sub-046e1343-dhgj9\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.570062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.569980    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:02.945879 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:02.945840    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"]
Oct 30 19:26:02.952177 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:26:02.952148    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99e607b_6d95_4df7_bd23_3f1ab968b1c5.slice/crio-f0dd08b701eeb0d6e6ec48c851cea8f91d289f20b9bf38a7084c2c96f5b0d463 WatchSource:0}: Error finding container f0dd08b701eeb0d6e6ec48c851cea8f91d289f20b9bf38a7084c2c96f5b0d463: Status 404 returned error can't find the container with id f0dd08b701eeb0d6e6ec48c851cea8f91d289f20b9bf38a7084c2c96f5b0d463
Oct 30 19:26:03.371262 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:03.371184    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9" event={"ID":"e99e607b-6d95-4df7-bd23-3f1ab968b1c5","Type":"ContainerStarted","Data":"5300b95f8a1cd1569a302635d306c84a1a0f8807036abfb8ea22a5ae3f1e1b93"}
Oct 30 19:26:03.371262 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:03.371234    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9" event={"ID":"e99e607b-6d95-4df7-bd23-3f1ab968b1c5","Type":"ContainerStarted","Data":"f0dd08b701eeb0d6e6ec48c851cea8f91d289f20b9bf38a7084c2c96f5b0d463"}
Oct 30 19:26:03.484877 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:03.484800    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9" podStartSLOduration=1.484785561 podStartE2EDuration="1.484785561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:03.483990484 +0000 UTC m=+2854.371907573" watchObservedRunningTime="2026-10-30 19:26:03.484785561 +0000 UTC m=+2854.372702650"
Oct 30 19:26:04.378233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:04.378196    4267 generic.go:356] "Generic (PLEG): container finished" podID="e99e607b-6d95-4df7-bd23-3f1ab968b1c5" containerID="5300b95f8a1cd1569a302635d306c84a1a0f8807036abfb8ea22a5ae3f1e1b93" exitCode=0
Oct 30 19:26:04.378670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:04.378278    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9" event={"ID":"e99e607b-6d95-4df7-bd23-3f1ab968b1c5","Type":"ContainerDied","Data":"5300b95f8a1cd1569a302635d306c84a1a0f8807036abfb8ea22a5ae3f1e1b93"}
Oct 30 19:26:05.512185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:05.512161    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:05.539225 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:05.539199    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-kube-api-access-xxt4h\" (UniqueName: \"kubernetes.io/projected/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-kube-api-access-xxt4h\") pod \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") "
Oct 30 19:26:05.539346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:05.539249    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-xks-ca\") pod \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") "
Oct 30 19:26:05.539346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:05.539279    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-openshift-ca\") pod \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\" (UID: \"e99e607b-6d95-4df7-bd23-3f1ab968b1c5\") "
Oct 30 19:26:05.539694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:05.539669    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-openshift-ca" pod "e99e607b-6d95-4df7-bd23-3f1ab968b1c5" (UID: "e99e607b-6d95-4df7-bd23-3f1ab968b1c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:05.541304 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:05.541274    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-xks-ca" pod "e99e607b-6d95-4df7-bd23-3f1ab968b1c5" (UID: "e99e607b-6d95-4df7-bd23-3f1ab968b1c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:05.541410 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:05.541325    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-kube-api-access-xxt4h" pod "e99e607b-6d95-4df7-bd23-3f1ab968b1c5" (UID: "e99e607b-6d95-4df7-bd23-3f1ab968b1c5"). InnerVolumeSpecName "kube-api-access-xxt4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:05.640453 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:05.640381    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxt4h\" (UniqueName: \"kubernetes.io/projected/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-kube-api-access-xxt4h\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:05.640453 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:05.640409    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:05.640453 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:05.640419    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99e607b-6d95-4df7-bd23-3f1ab968b1c5-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:06.389466 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:06.389429    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9" event={"ID":"e99e607b-6d95-4df7-bd23-3f1ab968b1c5","Type":"ContainerDied","Data":"f0dd08b701eeb0d6e6ec48c851cea8f91d289f20b9bf38a7084c2c96f5b0d463"}
Oct 30 19:26:06.389659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:06.389488    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0dd08b701eeb0d6e6ec48c851cea8f91d289f20b9bf38a7084c2c96f5b0d463"
Oct 30 19:26:06.389659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:06.389524    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"
Oct 30 19:26:12.901343 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:12.901305    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"]
Oct 30 19:26:12.902016 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:12.901736    4267 cpu_manager.go:414] "containerMap: removing container" podUID="e99e607b-6d95-4df7-bd23-3f1ab968b1c5" containerName="revoke-keys"
Oct 30 19:26:12.902016 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:12.901750    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99e607b-6d95-4df7-bd23-3f1ab968b1c5" containerName="revoke-keys"
Oct 30 19:26:12.902016 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:12.901822    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99e607b-6d95-4df7-bd23-3f1ab968b1c5" containerName="revoke-keys"
Oct 30 19:26:12.905000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:12.904972    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:12.960276 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:12.960241    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:26:13.007214 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.007179    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:13.007214 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.007213    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:26:13.007440 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.007258    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:26:13.009675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.009645    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-9rhbz\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:13.010000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.009683    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7xq\" (UniqueName: \"kubernetes.io/projected/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-kube-api-access-6z7xq\") pod \"maas-api-revoke-sub-d3fba056-9rhbz\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:13.010000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.009808    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-9rhbz\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:13.010000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.009991    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"]
Oct 30 19:26:13.111086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.111042    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-9rhbz\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:13.111295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.111128    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-9rhbz\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:13.111295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.111166    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7xq\" (UniqueName: \"kubernetes.io/projected/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-kube-api-access-6z7xq\") pod \"maas-api-revoke-sub-d3fba056-9rhbz\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:13.111912 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.111886    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-9rhbz\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:13.113614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.113589    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-9rhbz\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:13.168306 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.168223    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7xq\" (UniqueName: \"kubernetes.io/projected/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-kube-api-access-6z7xq\") pod \"maas-api-revoke-sub-d3fba056-9rhbz\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:13.217572 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.217541    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:13.646510 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:13.641426    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"]
Oct 30 19:26:13.650670 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:26:13.650635    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2fba27_0b4b_4047_bbc2_b5539b32a13a.slice/crio-a02e70d9e36631466b609ead80575795cc1f174840994ed30b9ec1604d5d69a6 WatchSource:0}: Error finding container a02e70d9e36631466b609ead80575795cc1f174840994ed30b9ec1604d5d69a6: Status 404 returned error can't find the container with id a02e70d9e36631466b609ead80575795cc1f174840994ed30b9ec1604d5d69a6
Oct 30 19:26:14.456515 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:14.456438    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz" event={"ID":"fa2fba27-0b4b-4047-bbc2-b5539b32a13a","Type":"ContainerStarted","Data":"a642752498c492edc04029d8155c6aaf5d36d476c8460779613497b5a532f4ef"}
Oct 30 19:26:14.457066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:14.456531    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz" event={"ID":"fa2fba27-0b4b-4047-bbc2-b5539b32a13a","Type":"ContainerStarted","Data":"a02e70d9e36631466b609ead80575795cc1f174840994ed30b9ec1604d5d69a6"}
Oct 30 19:26:14.582295 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:14.582241    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz" podStartSLOduration=2.582226371 podStartE2EDuration="2.582226371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:14.580904219 +0000 UTC m=+2865.468821311" watchObservedRunningTime="2026-10-30 19:26:14.582226371 +0000 UTC m=+2865.470143459"
Oct 30 19:26:15.468721 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:15.468679    4267 generic.go:356] "Generic (PLEG): container finished" podID="fa2fba27-0b4b-4047-bbc2-b5539b32a13a" containerID="a642752498c492edc04029d8155c6aaf5d36d476c8460779613497b5a532f4ef" exitCode=0
Oct 30 19:26:15.469275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:15.468901    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz" event={"ID":"fa2fba27-0b4b-4047-bbc2-b5539b32a13a","Type":"ContainerDied","Data":"a642752498c492edc04029d8155c6aaf5d36d476c8460779613497b5a532f4ef"}
Oct 30 19:26:17.134942 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.134621    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:17.185924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.182791    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-openshift-ca\") pod \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") "
Oct 30 19:26:17.185924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.182921    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-kube-api-access-6z7xq\" (UniqueName: \"kubernetes.io/projected/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-kube-api-access-6z7xq\") pod \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") "
Oct 30 19:26:17.185924 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.182965    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-xks-ca\") pod \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\" (UID: \"fa2fba27-0b4b-4047-bbc2-b5539b32a13a\") "
Oct 30 19:26:17.186751 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.186381    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-openshift-ca" pod "fa2fba27-0b4b-4047-bbc2-b5539b32a13a" (UID: "fa2fba27-0b4b-4047-bbc2-b5539b32a13a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:17.201646 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.201516    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-kube-api-access-6z7xq" pod "fa2fba27-0b4b-4047-bbc2-b5539b32a13a" (UID: "fa2fba27-0b4b-4047-bbc2-b5539b32a13a"). InnerVolumeSpecName "kube-api-access-6z7xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:17.201646 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.201611    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-xks-ca" pod "fa2fba27-0b4b-4047-bbc2-b5539b32a13a" (UID: "fa2fba27-0b4b-4047-bbc2-b5539b32a13a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:17.292156 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.292062    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:17.292156 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.292105    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z7xq\" (UniqueName: \"kubernetes.io/projected/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-kube-api-access-6z7xq\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:17.292156 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.292124    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2fba27-0b4b-4047-bbc2-b5539b32a13a-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:17.530534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.526056    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz" event={"ID":"fa2fba27-0b4b-4047-bbc2-b5539b32a13a","Type":"ContainerDied","Data":"a02e70d9e36631466b609ead80575795cc1f174840994ed30b9ec1604d5d69a6"}
Oct 30 19:26:17.530534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.526121    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a02e70d9e36631466b609ead80575795cc1f174840994ed30b9ec1604d5d69a6"
Oct 30 19:26:17.530534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.526196    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"
Oct 30 19:26:17.900674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.900625    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8cp28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:26:17.901003 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:17.900965    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podUID="b4b5cee2-4ba7-4d14-bde2-dd266f4706f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:19.045450 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:19.045407    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"]
Oct 30 19:26:19.095375 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:19.095340    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cae1c75-sm46n"]
Oct 30 19:26:19.219656 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:19.219625    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400aabd2-4d58-44f7-959f-aeb01952b4b3" path="/var/lib/kubelet/pods/400aabd2-4d58-44f7-959f-aeb01952b4b3/volumes"
Oct 30 19:26:20.158152 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:26:20.158115    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:26:32.167698 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:32.167659    4267 scope.go:122] "RemoveContainer" containerID="4f46c69102443f81c62b949e523993f2bacbe392612c0b689062b633f68d6a94"
Oct 30 19:26:32.236517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:32.230112    4267 scope.go:122] "RemoveContainer" containerID="5a0a27d57015050f1f56f6436786beeed1e96783ae055584c980bb0741f68e25"
Oct 30 19:26:32.256648 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:32.256618    4267 scope.go:122] "RemoveContainer" containerID="1b20d739a74b1fc8c59f8be672e8dbb274d2f76e48f85436cf90095485716bc7"
Oct 30 19:26:32.304952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:32.304927    4267 scope.go:122] "RemoveContainer" containerID="c540b4c67efe7aa334f7a2cb2666069a8d6085c257afaaab258bd7f49fba070e"
Oct 30 19:26:32.368171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:32.368129    4267 scope.go:122] "RemoveContainer" containerID="5f645f22a658602a3bf6eb40483f2117a9f6c4282f117a71437660947ab8855e"
Oct 30 19:26:32.881663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:32.881620    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8cp28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:26:32.882005 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:32.881970    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podUID="b4b5cee2-4ba7-4d14-bde2-dd266f4706f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:47.743373 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.743333    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"]
Oct 30 19:26:47.743895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.743740    4267 cpu_manager.go:414] "containerMap: removing container" podUID="fa2fba27-0b4b-4047-bbc2-b5539b32a13a" containerName="revoke-keys"
Oct 30 19:26:47.743895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.743753    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2fba27-0b4b-4047-bbc2-b5539b32a13a" containerName="revoke-keys"
Oct 30 19:26:47.743895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.743808    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2fba27-0b4b-4047-bbc2-b5539b32a13a" containerName="revoke-keys"
Oct 30 19:26:47.746882 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.746861    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:47.800212 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.800176    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:26:47.849845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.849806    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:47.850029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.849893    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:26:47.850029 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.849934    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:26:47.850131 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.850110    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9sm69\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:47.850170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.850142    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ghl\" (UniqueName: \"kubernetes.io/projected/c92449a5-20b6-444b-aa39-a69c029a4eac-kube-api-access-r5ghl\") pod \"maas-api-revoke-sub-ae594979-9sm69\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:47.850314 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.850291    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9sm69\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:47.852603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.852583    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"]
Oct 30 19:26:47.951160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.951123    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9sm69\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:47.951160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.951163    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ghl\" (UniqueName: \"kubernetes.io/projected/c92449a5-20b6-444b-aa39-a69c029a4eac-kube-api-access-r5ghl\") pod \"maas-api-revoke-sub-ae594979-9sm69\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:47.951390 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.951220    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9sm69\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:47.951997 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.951967    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-9sm69\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:47.954081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:47.954053    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-9sm69\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:48.008884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:48.008803    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ghl\" (UniqueName: \"kubernetes.io/projected/c92449a5-20b6-444b-aa39-a69c029a4eac-kube-api-access-r5ghl\") pod \"maas-api-revoke-sub-ae594979-9sm69\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:48.064963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:48.064928    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:48.434546 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:48.434520    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"]
Oct 30 19:26:48.436425 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:26:48.436395    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92449a5_20b6_444b_aa39_a69c029a4eac.slice/crio-5567cd8c1c9e8549e55f3ec8300be210850bcf9a28531c291591309976cd2cae WatchSource:0}: Error finding container 5567cd8c1c9e8549e55f3ec8300be210850bcf9a28531c291591309976cd2cae: Status 404 returned error can't find the container with id 5567cd8c1c9e8549e55f3ec8300be210850bcf9a28531c291591309976cd2cae
Oct 30 19:26:48.751952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:48.751917    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69" event={"ID":"c92449a5-20b6-444b-aa39-a69c029a4eac","Type":"ContainerStarted","Data":"7aabc826dea85ef1007dca6aa0320056f43fb40e3bb3336018315eef94153433"}
Oct 30 19:26:48.752336 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:48.751978    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69" event={"ID":"c92449a5-20b6-444b-aa39-a69c029a4eac","Type":"ContainerStarted","Data":"5567cd8c1c9e8549e55f3ec8300be210850bcf9a28531c291591309976cd2cae"}
Oct 30 19:26:48.868931 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:48.868868    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69" podStartSLOduration=1.868853234 podStartE2EDuration="1.868853234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:48.866372706 +0000 UTC m=+2899.754289796" watchObservedRunningTime="2026-10-30 19:26:48.868853234 +0000 UTC m=+2899.756770320"
Oct 30 19:26:49.758217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:49.758180    4267 generic.go:356] "Generic (PLEG): container finished" podID="c92449a5-20b6-444b-aa39-a69c029a4eac" containerID="7aabc826dea85ef1007dca6aa0320056f43fb40e3bb3336018315eef94153433" exitCode=0
Oct 30 19:26:49.758671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:49.758271    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69" event={"ID":"c92449a5-20b6-444b-aa39-a69c029a4eac","Type":"ContainerDied","Data":"7aabc826dea85ef1007dca6aa0320056f43fb40e3bb3336018315eef94153433"}
Oct 30 19:26:50.157381 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:26:50.157300    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:26:50.898658 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:50.898632    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:50.981471 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:50.981432    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-xks-ca\") pod \"c92449a5-20b6-444b-aa39-a69c029a4eac\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") "
Oct 30 19:26:50.981699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:50.981536    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-openshift-ca\") pod \"c92449a5-20b6-444b-aa39-a69c029a4eac\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") "
Oct 30 19:26:50.981699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:50.981576    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92449a5-20b6-444b-aa39-a69c029a4eac-kube-api-access-r5ghl\" (UniqueName: \"kubernetes.io/projected/c92449a5-20b6-444b-aa39-a69c029a4eac-kube-api-access-r5ghl\") pod \"c92449a5-20b6-444b-aa39-a69c029a4eac\" (UID: \"c92449a5-20b6-444b-aa39-a69c029a4eac\") "
Oct 30 19:26:50.981907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:50.981881    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-openshift-ca" pod "c92449a5-20b6-444b-aa39-a69c029a4eac" (UID: "c92449a5-20b6-444b-aa39-a69c029a4eac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:50.983681 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:50.983656    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-xks-ca" pod "c92449a5-20b6-444b-aa39-a69c029a4eac" (UID: "c92449a5-20b6-444b-aa39-a69c029a4eac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:50.983862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:50.983842    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92449a5-20b6-444b-aa39-a69c029a4eac-kube-api-access-r5ghl" pod "c92449a5-20b6-444b-aa39-a69c029a4eac" (UID: "c92449a5-20b6-444b-aa39-a69c029a4eac"). InnerVolumeSpecName "kube-api-access-r5ghl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:51.083279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:51.083195    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:51.083279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:51.083222    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5ghl\" (UniqueName: \"kubernetes.io/projected/c92449a5-20b6-444b-aa39-a69c029a4eac-kube-api-access-r5ghl\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:51.083279 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:51.083232    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c92449a5-20b6-444b-aa39-a69c029a4eac-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:51.770299 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:51.770271    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"
Oct 30 19:26:51.770454 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:51.770315    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69" event={"ID":"c92449a5-20b6-444b-aa39-a69c029a4eac","Type":"ContainerDied","Data":"5567cd8c1c9e8549e55f3ec8300be210850bcf9a28531c291591309976cd2cae"}
Oct 30 19:26:51.770454 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:26:51.770366    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5567cd8c1c9e8549e55f3ec8300be210850bcf9a28531c291591309976cd2cae"
Oct 30 19:27:16.032699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.032619    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"]
Oct 30 19:27:16.033172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.033026    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c92449a5-20b6-444b-aa39-a69c029a4eac" containerName="revoke-keys"
Oct 30 19:27:16.033172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.033041    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92449a5-20b6-444b-aa39-a69c029a4eac" containerName="revoke-keys"
Oct 30 19:27:16.033172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.033117    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92449a5-20b6-444b-aa39-a69c029a4eac" containerName="revoke-keys"
Oct 30 19:27:16.036528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.036486    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.092768 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.092735    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:16.092944 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.092768    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:27:16.092944 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.092736    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:27:16.092944 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.092823    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:27:16.145174 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.145141    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"]
Oct 30 19:27:16.176349 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.176321    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9f8b\" (UniqueName: \"kubernetes.io/projected/84f8604f-6f86-4e10-9bef-405bea72e7e4-kube-api-access-j9f8b\") pod \"maas-api-revoke-sub-a4aef7d7-gtflg\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.176554 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.176356    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gtflg\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.176554 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.176480    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gtflg\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.277273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.277237    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9f8b\" (UniqueName: \"kubernetes.io/projected/84f8604f-6f86-4e10-9bef-405bea72e7e4-kube-api-access-j9f8b\") pod \"maas-api-revoke-sub-a4aef7d7-gtflg\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.277273 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.277277    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gtflg\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.277528 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.277315    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gtflg\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.278004 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.277976    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gtflg\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.279793 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.279764    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gtflg\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.334605 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.334533    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9f8b\" (UniqueName: \"kubernetes.io/projected/84f8604f-6f86-4e10-9bef-405bea72e7e4-kube-api-access-j9f8b\") pod \"maas-api-revoke-sub-a4aef7d7-gtflg\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.349363 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.349338    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:16.719841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.719811    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"]
Oct 30 19:27:16.721098 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:27:16.721071    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f8604f_6f86_4e10_9bef_405bea72e7e4.slice/crio-91ed7760b8df0f9e619c79fac9b6f3cde35e475c3c39e8b6f4843152a5867e29 WatchSource:0}: Error finding container 91ed7760b8df0f9e619c79fac9b6f3cde35e475c3c39e8b6f4843152a5867e29: Status 404 returned error can't find the container with id 91ed7760b8df0f9e619c79fac9b6f3cde35e475c3c39e8b6f4843152a5867e29
Oct 30 19:27:16.979310 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:16.979235    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg" event={"ID":"84f8604f-6f86-4e10-9bef-405bea72e7e4","Type":"ContainerStarted","Data":"91ed7760b8df0f9e619c79fac9b6f3cde35e475c3c39e8b6f4843152a5867e29"}
Oct 30 19:27:17.986562 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:17.986529    4267 generic.go:356] "Generic (PLEG): container finished" podID="84f8604f-6f86-4e10-9bef-405bea72e7e4" containerID="a451ee609e36b9e6e7ee33c41d159cb15d968d0d6d84f06f649db3f2b5f82d92" exitCode=0
Oct 30 19:27:17.986990 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:17.986621    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg" event={"ID":"84f8604f-6f86-4e10-9bef-405bea72e7e4","Type":"ContainerDied","Data":"a451ee609e36b9e6e7ee33c41d159cb15d968d0d6d84f06f649db3f2b5f82d92"}
Oct 30 19:27:19.127478 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.127440    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:19.307119 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.307029    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-openshift-ca\") pod \"84f8604f-6f86-4e10-9bef-405bea72e7e4\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") "
Oct 30 19:27:19.307285 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.307227    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-xks-ca\") pod \"84f8604f-6f86-4e10-9bef-405bea72e7e4\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") "
Oct 30 19:27:19.307336 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.307287    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f8604f-6f86-4e10-9bef-405bea72e7e4-kube-api-access-j9f8b\" (UniqueName: \"kubernetes.io/projected/84f8604f-6f86-4e10-9bef-405bea72e7e4-kube-api-access-j9f8b\") pod \"84f8604f-6f86-4e10-9bef-405bea72e7e4\" (UID: \"84f8604f-6f86-4e10-9bef-405bea72e7e4\") "
Oct 30 19:27:19.307423 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.307396    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-openshift-ca" pod "84f8604f-6f86-4e10-9bef-405bea72e7e4" (UID: "84f8604f-6f86-4e10-9bef-405bea72e7e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:19.309422 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.309394    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f8604f-6f86-4e10-9bef-405bea72e7e4-kube-api-access-j9f8b" pod "84f8604f-6f86-4e10-9bef-405bea72e7e4" (UID: "84f8604f-6f86-4e10-9bef-405bea72e7e4"). InnerVolumeSpecName "kube-api-access-j9f8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:19.309552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.309417    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-xks-ca" pod "84f8604f-6f86-4e10-9bef-405bea72e7e4" (UID: "84f8604f-6f86-4e10-9bef-405bea72e7e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:19.408503 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.408463    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:27:19.408657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.408528    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9f8b\" (UniqueName: \"kubernetes.io/projected/84f8604f-6f86-4e10-9bef-405bea72e7e4-kube-api-access-j9f8b\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:27:19.408657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.408544    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f8604f-6f86-4e10-9bef-405bea72e7e4-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:27:19.998209 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.998177    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"
Oct 30 19:27:19.998401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.998185    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg" event={"ID":"84f8604f-6f86-4e10-9bef-405bea72e7e4","Type":"ContainerDied","Data":"91ed7760b8df0f9e619c79fac9b6f3cde35e475c3c39e8b6f4843152a5867e29"}
Oct 30 19:27:19.998401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:19.998316    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91ed7760b8df0f9e619c79fac9b6f3cde35e475c3c39e8b6f4843152a5867e29"
Oct 30 19:27:20.158302 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:27:20.158262    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:27:32.669041 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:32.669002    4267 scope.go:122] "RemoveContainer" containerID="2b3bfe5149e4d6601c6c79433a1f5bee19b8c1ce661a1809b672afe86307298f"
Oct 30 19:27:32.678037 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:32.678014    4267 scope.go:122] "RemoveContainer" containerID="536dd7f7697f75400f72cda01232b5b51d6abe61e4199a90699c7ce2a1c0282e"
Oct 30 19:27:46.352781 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.352741    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"]
Oct 30 19:27:46.353186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.353160    4267 cpu_manager.go:414] "containerMap: removing container" podUID="84f8604f-6f86-4e10-9bef-405bea72e7e4" containerName="revoke-keys"
Oct 30 19:27:46.353186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.353174    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f8604f-6f86-4e10-9bef-405bea72e7e4" containerName="revoke-keys"
Oct 30 19:27:46.353276 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.353266    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f8604f-6f86-4e10-9bef-405bea72e7e4" containerName="revoke-keys"
Oct 30 19:27:46.357876 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.357856    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:46.461149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.461096    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:27:46.461342 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.461297    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:27:46.461438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.461377    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:27:46.461438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.461403    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:46.463932 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.463901    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"]
Oct 30 19:27:46.503933 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.503887    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6q4b9\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:46.504128 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.504008    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6q4b9\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:46.504128 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.504077    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znpqc\" (UniqueName: \"kubernetes.io/projected/78e8b7ce-7df7-4a36-adf8-42994c032319-kube-api-access-znpqc\") pod \"maas-api-revoke-sub-79805ca9-6q4b9\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:46.605560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.605440    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6q4b9\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:46.605560 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.605536    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6q4b9\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:46.605770 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.605578    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpqc\" (UniqueName: \"kubernetes.io/projected/78e8b7ce-7df7-4a36-adf8-42994c032319-kube-api-access-znpqc\") pod \"maas-api-revoke-sub-79805ca9-6q4b9\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:46.606255 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.606230    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6q4b9\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:46.607966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.607945    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6q4b9\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:46.664191 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.664162    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znpqc\" (UniqueName: \"kubernetes.io/projected/78e8b7ce-7df7-4a36-adf8-42994c032319-kube-api-access-znpqc\") pod \"maas-api-revoke-sub-79805ca9-6q4b9\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:46.671214 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:46.671190    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:47.046466 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:47.046432    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"]
Oct 30 19:27:47.046956 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:27:47.046927    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e8b7ce_7df7_4a36_adf8_42994c032319.slice/crio-75b981a76d686763b14fb2805e0cc0744ee9678180d6d3a9b0d7e16c6de53a27 WatchSource:0}: Error finding container 75b981a76d686763b14fb2805e0cc0744ee9678180d6d3a9b0d7e16c6de53a27: Status 404 returned error can't find the container with id 75b981a76d686763b14fb2805e0cc0744ee9678180d6d3a9b0d7e16c6de53a27
Oct 30 19:27:47.242324 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:47.242297    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9" event={"ID":"78e8b7ce-7df7-4a36-adf8-42994c032319","Type":"ContainerStarted","Data":"75b981a76d686763b14fb2805e0cc0744ee9678180d6d3a9b0d7e16c6de53a27"}
Oct 30 19:27:48.248930 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:48.248893    4267 generic.go:356] "Generic (PLEG): container finished" podID="78e8b7ce-7df7-4a36-adf8-42994c032319" containerID="18f8483ac36d880f8002ba922af3a6ee3c6b559edee27dd16ff0b84f57771bf8" exitCode=0
Oct 30 19:27:48.249353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:48.248963    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9" event={"ID":"78e8b7ce-7df7-4a36-adf8-42994c032319","Type":"ContainerDied","Data":"18f8483ac36d880f8002ba922af3a6ee3c6b559edee27dd16ff0b84f57771bf8"}
Oct 30 19:27:49.387367 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:49.387343    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:49.535832 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:49.535730    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-openshift-ca\") pod \"78e8b7ce-7df7-4a36-adf8-42994c032319\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") "
Oct 30 19:27:49.536046 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:49.535891    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-xks-ca\") pod \"78e8b7ce-7df7-4a36-adf8-42994c032319\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") "
Oct 30 19:27:49.536046 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:49.535913    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e8b7ce-7df7-4a36-adf8-42994c032319-kube-api-access-znpqc\" (UniqueName: \"kubernetes.io/projected/78e8b7ce-7df7-4a36-adf8-42994c032319-kube-api-access-znpqc\") pod \"78e8b7ce-7df7-4a36-adf8-42994c032319\" (UID: \"78e8b7ce-7df7-4a36-adf8-42994c032319\") "
Oct 30 19:27:49.536178 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:49.536136    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-openshift-ca" pod "78e8b7ce-7df7-4a36-adf8-42994c032319" (UID: "78e8b7ce-7df7-4a36-adf8-42994c032319"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:49.538013 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:49.537987    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-xks-ca" pod "78e8b7ce-7df7-4a36-adf8-42994c032319" (UID: "78e8b7ce-7df7-4a36-adf8-42994c032319"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:49.538160 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:49.538017    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e8b7ce-7df7-4a36-adf8-42994c032319-kube-api-access-znpqc" pod "78e8b7ce-7df7-4a36-adf8-42994c032319" (UID: "78e8b7ce-7df7-4a36-adf8-42994c032319"). InnerVolumeSpecName "kube-api-access-znpqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:49.636783 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:49.636735    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:27:49.636783 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:49.636778    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e8b7ce-7df7-4a36-adf8-42994c032319-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:27:49.636783 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:49.636789    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znpqc\" (UniqueName: \"kubernetes.io/projected/78e8b7ce-7df7-4a36-adf8-42994c032319-kube-api-access-znpqc\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:27:50.158453 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:27:50.158409    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:27:50.260912 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:50.260883    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"
Oct 30 19:27:50.261105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:50.260921    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9" event={"ID":"78e8b7ce-7df7-4a36-adf8-42994c032319","Type":"ContainerDied","Data":"75b981a76d686763b14fb2805e0cc0744ee9678180d6d3a9b0d7e16c6de53a27"}
Oct 30 19:27:50.261105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:50.260972    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75b981a76d686763b14fb2805e0cc0744ee9678180d6d3a9b0d7e16c6de53a27"
Oct 30 19:27:55.059842 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:55.059805    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"]
Oct 30 19:27:55.110175 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:55.110138    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59199a5a-qpv2d"]
Oct 30 19:27:55.218603 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:27:55.218567    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1b4aa0-90f3-4b8b-a627-c0619805940f" path="/var/lib/kubelet/pods/8a1b4aa0-90f3-4b8b-a627-c0619805940f/volumes"
Oct 30 19:28:06.996694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:06.996653    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"]
Oct 30 19:28:06.997102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:06.997041    4267 cpu_manager.go:414] "containerMap: removing container" podUID="78e8b7ce-7df7-4a36-adf8-42994c032319" containerName="revoke-keys"
Oct 30 19:28:06.997102 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:06.997054    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e8b7ce-7df7-4a36-adf8-42994c032319" containerName="revoke-keys"
Oct 30 19:28:06.997186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:06.997161    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e8b7ce-7df7-4a36-adf8-42994c032319" containerName="revoke-keys"
Oct 30 19:28:07.000348 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.000329    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.054368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.054321    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:07.057979 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.057945    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"]
Oct 30 19:28:07.059655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.059629    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324636a4-6291-44ac-bab9-90c212614090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s56dh\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.059797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.059761    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54pdp\" (UniqueName: \"kubernetes.io/projected/324636a4-6291-44ac-bab9-90c212614090-kube-api-access-54pdp\") pod \"maas-api-revoke-sub-0b7e8edb-s56dh\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.059797 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.059780    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324636a4-6291-44ac-bab9-90c212614090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s56dh\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.104130 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.104099    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:28:07.104130 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.104112    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:28:07.104130 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.104130    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:28:07.160897 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.160851    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54pdp\" (UniqueName: \"kubernetes.io/projected/324636a4-6291-44ac-bab9-90c212614090-kube-api-access-54pdp\") pod \"maas-api-revoke-sub-0b7e8edb-s56dh\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.160897 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.160899    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324636a4-6291-44ac-bab9-90c212614090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s56dh\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.161121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.160939    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324636a4-6291-44ac-bab9-90c212614090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s56dh\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.161794 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.161767    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324636a4-6291-44ac-bab9-90c212614090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s56dh\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.163992 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.163962    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324636a4-6291-44ac-bab9-90c212614090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s56dh\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.220747 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.220720    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54pdp\" (UniqueName: \"kubernetes.io/projected/324636a4-6291-44ac-bab9-90c212614090-kube-api-access-54pdp\") pod \"maas-api-revoke-sub-0b7e8edb-s56dh\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.313267 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.313184    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:07.688282 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:07.688249    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"]
Oct 30 19:28:07.689429 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:28:07.689404    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324636a4_6291_44ac_bab9_90c212614090.slice/crio-604a438e79ac4da6c577938a81ecdb5f624d1a11a1b5280a67a239563b675578 WatchSource:0}: Error finding container 604a438e79ac4da6c577938a81ecdb5f624d1a11a1b5280a67a239563b675578: Status 404 returned error can't find the container with id 604a438e79ac4da6c577938a81ecdb5f624d1a11a1b5280a67a239563b675578
Oct 30 19:28:08.388744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:08.388653    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh" event={"ID":"324636a4-6291-44ac-bab9-90c212614090","Type":"ContainerStarted","Data":"925453b4c674d04bff5d80465a14a61da520c315f39ba10e77fbd1e3875f806b"}
Oct 30 19:28:08.388744 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:08.388719    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh" event={"ID":"324636a4-6291-44ac-bab9-90c212614090","Type":"ContainerStarted","Data":"604a438e79ac4da6c577938a81ecdb5f624d1a11a1b5280a67a239563b675578"}
Oct 30 19:28:08.509064 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:08.509008    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh" podStartSLOduration=2.50899155 podStartE2EDuration="2.50899155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:08.507097332 +0000 UTC m=+2979.395014422" watchObservedRunningTime="2026-10-30 19:28:08.50899155 +0000 UTC m=+2979.396908639"
Oct 30 19:28:09.395469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:09.395426    4267 generic.go:356] "Generic (PLEG): container finished" podID="324636a4-6291-44ac-bab9-90c212614090" containerID="925453b4c674d04bff5d80465a14a61da520c315f39ba10e77fbd1e3875f806b" exitCode=0
Oct 30 19:28:09.395922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:09.395533    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh" event={"ID":"324636a4-6291-44ac-bab9-90c212614090","Type":"ContainerDied","Data":"925453b4c674d04bff5d80465a14a61da520c315f39ba10e77fbd1e3875f806b"}
Oct 30 19:28:10.535404 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:10.535377    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:10.586607 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:10.586573    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324636a4-6291-44ac-bab9-90c212614090-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324636a4-6291-44ac-bab9-90c212614090-maas-api-xks-ca\") pod \"324636a4-6291-44ac-bab9-90c212614090\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") "
Oct 30 19:28:10.586607 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:10.586618    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324636a4-6291-44ac-bab9-90c212614090-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324636a4-6291-44ac-bab9-90c212614090-maas-api-openshift-ca\") pod \"324636a4-6291-44ac-bab9-90c212614090\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") "
Oct 30 19:28:10.586828 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:10.586682    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324636a4-6291-44ac-bab9-90c212614090-kube-api-access-54pdp\" (UniqueName: \"kubernetes.io/projected/324636a4-6291-44ac-bab9-90c212614090-kube-api-access-54pdp\") pod \"324636a4-6291-44ac-bab9-90c212614090\" (UID: \"324636a4-6291-44ac-bab9-90c212614090\") "
Oct 30 19:28:10.587074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:10.587048    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324636a4-6291-44ac-bab9-90c212614090-maas-api-openshift-ca" pod "324636a4-6291-44ac-bab9-90c212614090" (UID: "324636a4-6291-44ac-bab9-90c212614090"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:10.588843 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:10.588817    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324636a4-6291-44ac-bab9-90c212614090-maas-api-xks-ca" pod "324636a4-6291-44ac-bab9-90c212614090" (UID: "324636a4-6291-44ac-bab9-90c212614090"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:10.588935 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:10.588855    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324636a4-6291-44ac-bab9-90c212614090-kube-api-access-54pdp" pod "324636a4-6291-44ac-bab9-90c212614090" (UID: "324636a4-6291-44ac-bab9-90c212614090"). InnerVolumeSpecName "kube-api-access-54pdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:10.687793 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:10.687759    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324636a4-6291-44ac-bab9-90c212614090-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:28:10.687793 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:10.687789    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324636a4-6291-44ac-bab9-90c212614090-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:28:10.687793 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:10.687800    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54pdp\" (UniqueName: \"kubernetes.io/projected/324636a4-6291-44ac-bab9-90c212614090-kube-api-access-54pdp\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:28:11.408008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:11.407981    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"
Oct 30 19:28:11.408228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:11.407980    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh" event={"ID":"324636a4-6291-44ac-bab9-90c212614090","Type":"ContainerDied","Data":"604a438e79ac4da6c577938a81ecdb5f624d1a11a1b5280a67a239563b675578"}
Oct 30 19:28:11.408383 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:11.408250    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="604a438e79ac4da6c577938a81ecdb5f624d1a11a1b5280a67a239563b675578"
Oct 30 19:28:18.311234 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.311189    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"]
Oct 30 19:28:18.311775 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.311751    4267 cpu_manager.go:414] "containerMap: removing container" podUID="324636a4-6291-44ac-bab9-90c212614090" containerName="revoke-keys"
Oct 30 19:28:18.311865 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.311778    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324636a4-6291-44ac-bab9-90c212614090" containerName="revoke-keys"
Oct 30 19:28:18.311922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.311868    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="324636a4-6291-44ac-bab9-90c212614090" containerName="revoke-keys"
Oct 30 19:28:18.315120 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.315099    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:18.349692 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.349659    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qxj5\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:18.349870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.349720    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvwnc\" (UniqueName: \"kubernetes.io/projected/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-kube-api-access-nvwnc\") pod \"maas-api-revoke-sub-3d4156f1-8qxj5\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:18.349921 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.349869    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qxj5\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:18.368909 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.368878    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:18.372537 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.372481    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"]
Oct 30 19:28:18.418695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.418659    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:28:18.418695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.418693    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:28:18.418907 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.418740    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:28:18.450885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.450856    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvwnc\" (UniqueName: \"kubernetes.io/projected/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-kube-api-access-nvwnc\") pod \"maas-api-revoke-sub-3d4156f1-8qxj5\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:18.451081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.450912    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qxj5\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:18.451081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.450941    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qxj5\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:18.451640 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.451616    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qxj5\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:18.453212 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.453193    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qxj5\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:18.507706 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.507676    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvwnc\" (UniqueName: \"kubernetes.io/projected/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-kube-api-access-nvwnc\") pod \"maas-api-revoke-sub-3d4156f1-8qxj5\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:18.627844 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:18.627767    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:19.002109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:19.002068    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"]
Oct 30 19:28:19.005629 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:28:19.005586    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc299bd53_cbfc_4d15_ab7a_0b5c8d3945d1.slice/crio-6e127f62cbc5266486b1c649585a8146e0282770298f3ba13a832f88e30a3258 WatchSource:0}: Error finding container 6e127f62cbc5266486b1c649585a8146e0282770298f3ba13a832f88e30a3258: Status 404 returned error can't find the container with id 6e127f62cbc5266486b1c649585a8146e0282770298f3ba13a832f88e30a3258
Oct 30 19:28:19.450656 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:19.450619    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5" event={"ID":"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1","Type":"ContainerStarted","Data":"2774c24698272c95c17de3f7464f9633e3bc7da32a042f82226278d849ea6614"}
Oct 30 19:28:19.451092 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:19.450677    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5" event={"ID":"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1","Type":"ContainerStarted","Data":"6e127f62cbc5266486b1c649585a8146e0282770298f3ba13a832f88e30a3258"}
Oct 30 19:28:19.567432 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:19.567331    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5" podStartSLOduration=1.5673176309999999 podStartE2EDuration="1.567317631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:19.566412403 +0000 UTC m=+2990.454329497" watchObservedRunningTime="2026-10-30 19:28:19.567317631 +0000 UTC m=+2990.455234719"
Oct 30 19:28:20.171177 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:28:20.171125    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:28:20.457682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:20.457646    4267 generic.go:356] "Generic (PLEG): container finished" podID="c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1" containerID="2774c24698272c95c17de3f7464f9633e3bc7da32a042f82226278d849ea6614" exitCode=0
Oct 30 19:28:20.458142 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:20.457707    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5" event={"ID":"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1","Type":"ContainerDied","Data":"2774c24698272c95c17de3f7464f9633e3bc7da32a042f82226278d849ea6614"}
Oct 30 19:28:21.594714 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:21.594689    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:21.678084 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:21.678041    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-openshift-ca\") pod \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") "
Oct 30 19:28:21.678272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:21.678108    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-xks-ca\") pod \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") "
Oct 30 19:28:21.678272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:21.678264    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-kube-api-access-nvwnc\" (UniqueName: \"kubernetes.io/projected/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-kube-api-access-nvwnc\") pod \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\" (UID: \"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1\") "
Oct 30 19:28:21.678481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:21.678452    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-openshift-ca" pod "c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1" (UID: "c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:21.680180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:21.680154    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-xks-ca" pod "c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1" (UID: "c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:21.680301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:21.680281    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-kube-api-access-nvwnc" pod "c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1" (UID: "c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1"). InnerVolumeSpecName "kube-api-access-nvwnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:21.779980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:21.779894    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:28:21.779980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:21.779927    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:28:21.779980 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:21.779936    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvwnc\" (UniqueName: \"kubernetes.io/projected/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1-kube-api-access-nvwnc\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:28:22.469228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:22.469192    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"
Oct 30 19:28:22.469228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:22.469220    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5" event={"ID":"c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1","Type":"ContainerDied","Data":"6e127f62cbc5266486b1c649585a8146e0282770298f3ba13a832f88e30a3258"}
Oct 30 19:28:22.469457 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:22.469273    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e127f62cbc5266486b1c649585a8146e0282770298f3ba13a832f88e30a3258"
Oct 30 19:28:32.706138 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:32.706106    4267 scope.go:122] "RemoveContainer" containerID="d061cd40c4924fbe679052e25fa39c1985c4f1d79ba17d1368d7df4911869b21"
Oct 30 19:28:32.715022 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:32.714990    4267 scope.go:122] "RemoveContainer" containerID="9c6e8ce136c5c038438aa149925237e08c5c61de2d627f2ff9ce2a0ed3c3bbb0"
Oct 30 19:28:36.254123 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.254086    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"]
Oct 30 19:28:36.303732 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.303692    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qxj5"]
Oct 30 19:28:36.374215 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.374185    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"]
Oct 30 19:28:36.374563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.374549    4267 cpu_manager.go:414] "containerMap: removing container" podUID="c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1" containerName="revoke-keys"
Oct 30 19:28:36.374625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.374564    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1" containerName="revoke-keys"
Oct 30 19:28:36.374674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.374643    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1" containerName="revoke-keys"
Oct 30 19:28:36.379193 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.379177    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:36.432061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.432026    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:36.432252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.432026    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:28:36.432252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.432032    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:28:36.432252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.432126    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:28:36.441887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.441858    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"]
Oct 30 19:28:36.523667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.523566    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-22bxl\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:36.523667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.523614    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-22bxl\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:36.523922 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.523763    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6wm\" (UniqueName: \"kubernetes.io/projected/fd33196f-87b6-4e7b-8489-d4a17be684ae-kube-api-access-rc6wm\") pod \"maas-api-revoke-sub-3d4156f1-22bxl\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:36.624582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.624541    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-22bxl\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:36.624778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.624592    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-22bxl\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:36.624778 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.624639    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6wm\" (UniqueName: \"kubernetes.io/projected/fd33196f-87b6-4e7b-8489-d4a17be684ae-kube-api-access-rc6wm\") pod \"maas-api-revoke-sub-3d4156f1-22bxl\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:36.625199 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.625177    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-22bxl\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:36.626951 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.626933    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-22bxl\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:36.682237 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.682198    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6wm\" (UniqueName: \"kubernetes.io/projected/fd33196f-87b6-4e7b-8489-d4a17be684ae-kube-api-access-rc6wm\") pod \"maas-api-revoke-sub-3d4156f1-22bxl\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:36.691404 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:36.691379    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:37.067269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:37.067244    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"]
Oct 30 19:28:37.068141 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:28:37.068115    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd33196f_87b6_4e7b_8489_d4a17be684ae.slice/crio-514c00d86c9218e17914c961284ec38c9786eb6cc9c02a635baba9fef5c8b912 WatchSource:0}: Error finding container 514c00d86c9218e17914c961284ec38c9786eb6cc9c02a635baba9fef5c8b912: Status 404 returned error can't find the container with id 514c00d86c9218e17914c961284ec38c9786eb6cc9c02a635baba9fef5c8b912
Oct 30 19:28:37.221175 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:37.221144    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1" path="/var/lib/kubelet/pods/c299bd53-cbfc-4d15-ab7a-0b5c8d3945d1/volumes"
Oct 30 19:28:37.551929 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:37.551886    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl" event={"ID":"fd33196f-87b6-4e7b-8489-d4a17be684ae","Type":"ContainerStarted","Data":"8bc8c35903b4781d7cf659ce5f096a0b27e69fffe5df7fd0a3e53ab0a6d02df3"}
Oct 30 19:28:37.552417 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:37.551942    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl" event={"ID":"fd33196f-87b6-4e7b-8489-d4a17be684ae","Type":"ContainerStarted","Data":"514c00d86c9218e17914c961284ec38c9786eb6cc9c02a635baba9fef5c8b912"}
Oct 30 19:28:37.666148 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:37.666054    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl" podStartSLOduration=1.6660400640000002 podStartE2EDuration="1.666040064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:37.665309721 +0000 UTC m=+3008.553226809" watchObservedRunningTime="2026-10-30 19:28:37.666040064 +0000 UTC m=+3008.553957172"
Oct 30 19:28:38.558730 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:38.558697    4267 generic.go:356] "Generic (PLEG): container finished" podID="fd33196f-87b6-4e7b-8489-d4a17be684ae" containerID="8bc8c35903b4781d7cf659ce5f096a0b27e69fffe5df7fd0a3e53ab0a6d02df3" exitCode=0
Oct 30 19:28:38.559140 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:38.558788    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl" event={"ID":"fd33196f-87b6-4e7b-8489-d4a17be684ae","Type":"ContainerDied","Data":"8bc8c35903b4781d7cf659ce5f096a0b27e69fffe5df7fd0a3e53ab0a6d02df3"}
Oct 30 19:28:39.697934 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:39.697907    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:39.752620 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:39.752587    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-openshift-ca\") pod \"fd33196f-87b6-4e7b-8489-d4a17be684ae\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") "
Oct 30 19:28:39.752795 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:39.752641    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-xks-ca\") pod \"fd33196f-87b6-4e7b-8489-d4a17be684ae\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") "
Oct 30 19:28:39.752795 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:39.752704    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd33196f-87b6-4e7b-8489-d4a17be684ae-kube-api-access-rc6wm\" (UniqueName: \"kubernetes.io/projected/fd33196f-87b6-4e7b-8489-d4a17be684ae-kube-api-access-rc6wm\") pod \"fd33196f-87b6-4e7b-8489-d4a17be684ae\" (UID: \"fd33196f-87b6-4e7b-8489-d4a17be684ae\") "
Oct 30 19:28:39.753058 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:39.753036    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-openshift-ca" pod "fd33196f-87b6-4e7b-8489-d4a17be684ae" (UID: "fd33196f-87b6-4e7b-8489-d4a17be684ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:39.754937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:39.754917    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd33196f-87b6-4e7b-8489-d4a17be684ae-kube-api-access-rc6wm" pod "fd33196f-87b6-4e7b-8489-d4a17be684ae" (UID: "fd33196f-87b6-4e7b-8489-d4a17be684ae"). InnerVolumeSpecName "kube-api-access-rc6wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:39.755015 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:39.754916    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-xks-ca" pod "fd33196f-87b6-4e7b-8489-d4a17be684ae" (UID: "fd33196f-87b6-4e7b-8489-d4a17be684ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:39.853742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:39.853657    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:28:39.853742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:39.853690    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd33196f-87b6-4e7b-8489-d4a17be684ae-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:28:39.853742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:39.853700    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6wm\" (UniqueName: \"kubernetes.io/projected/fd33196f-87b6-4e7b-8489-d4a17be684ae-kube-api-access-rc6wm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:28:40.570351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:40.570320    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"
Oct 30 19:28:40.570351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:40.570331    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl" event={"ID":"fd33196f-87b6-4e7b-8489-d4a17be684ae","Type":"ContainerDied","Data":"514c00d86c9218e17914c961284ec38c9786eb6cc9c02a635baba9fef5c8b912"}
Oct 30 19:28:40.570589 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:40.570378    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="514c00d86c9218e17914c961284ec38c9786eb6cc9c02a635baba9fef5c8b912"
Oct 30 19:28:50.157232 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:28:50.157142    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:28:51.049456 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:51.049414    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"]
Oct 30 19:28:51.098937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:51.098894    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-26tvl"]
Oct 30 19:28:51.218968 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:51.218937    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974108d4-dbc0-49f2-b013-80e7e8603a8f" path="/var/lib/kubelet/pods/974108d4-dbc0-49f2-b013-80e7e8603a8f/volumes"
Oct 30 19:28:57.744072 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.744033    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"]
Oct 30 19:28:57.780461 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.780427    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-22bxl"]
Oct 30 19:28:57.863663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.863625    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"]
Oct 30 19:28:57.864054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.864032    4267 cpu_manager.go:414] "containerMap: removing container" podUID="fd33196f-87b6-4e7b-8489-d4a17be684ae" containerName="revoke-keys"
Oct 30 19:28:57.864054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.864048    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd33196f-87b6-4e7b-8489-d4a17be684ae" containerName="revoke-keys"
Oct 30 19:28:57.864206 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.864106    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd33196f-87b6-4e7b-8489-d4a17be684ae" containerName="revoke-keys"
Oct 30 19:28:57.868313 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.868293    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:57.923137 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.923104    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:28:57.923137 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.923134    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:57.923368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.923138    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:28:57.923368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.923243    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:28:57.925663 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:57.925636    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"]
Oct 30 19:28:58.031848 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.031741    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621a91fe-7351-42c9-842f-a22388611929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f84m8\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:58.031848 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.031814    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621a91fe-7351-42c9-842f-a22388611929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f84m8\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:58.032069 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.031901    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzzx\" (UniqueName: \"kubernetes.io/projected/621a91fe-7351-42c9-842f-a22388611929-kube-api-access-rvzzx\") pod \"maas-api-revoke-sub-3d4156f1-f84m8\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:58.132829 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.132785    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621a91fe-7351-42c9-842f-a22388611929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f84m8\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:58.133050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.132867    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621a91fe-7351-42c9-842f-a22388611929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f84m8\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:58.133050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.132898    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzzx\" (UniqueName: \"kubernetes.io/projected/621a91fe-7351-42c9-842f-a22388611929-kube-api-access-rvzzx\") pod \"maas-api-revoke-sub-3d4156f1-f84m8\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:58.133600 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.133570    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621a91fe-7351-42c9-842f-a22388611929-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f84m8\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:58.135334 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.135310    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621a91fe-7351-42c9-842f-a22388611929-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f84m8\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:58.190555 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.190518    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzzx\" (UniqueName: \"kubernetes.io/projected/621a91fe-7351-42c9-842f-a22388611929-kube-api-access-rvzzx\") pod \"maas-api-revoke-sub-3d4156f1-f84m8\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:58.480929 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.480892    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:28:58.862265 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:58.862231    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"]
Oct 30 19:28:58.862979 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:28:58.862954    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621a91fe_7351_42c9_842f_a22388611929.slice/crio-5d5842f8a277a9a8197f26dbe9d500191f7bbbad424ba04b626b99a4d87f7bf6 WatchSource:0}: Error finding container 5d5842f8a277a9a8197f26dbe9d500191f7bbbad424ba04b626b99a4d87f7bf6: Status 404 returned error can't find the container with id 5d5842f8a277a9a8197f26dbe9d500191f7bbbad424ba04b626b99a4d87f7bf6
Oct 30 19:28:59.219463 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:59.219417    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd33196f-87b6-4e7b-8489-d4a17be684ae" path="/var/lib/kubelet/pods/fd33196f-87b6-4e7b-8489-d4a17be684ae/volumes"
Oct 30 19:28:59.692434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:59.692395    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8" event={"ID":"621a91fe-7351-42c9-842f-a22388611929","Type":"ContainerStarted","Data":"5e97569acdf7a67809e2ba38c6bb336986e70d1ca13e4225a9c5c43bd0094e06"}
Oct 30 19:28:59.692655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:59.692450    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8" event={"ID":"621a91fe-7351-42c9-842f-a22388611929","Type":"ContainerStarted","Data":"5d5842f8a277a9a8197f26dbe9d500191f7bbbad424ba04b626b99a4d87f7bf6"}
Oct 30 19:28:59.806615 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:28:59.806559    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8" podStartSLOduration=2.806544567 podStartE2EDuration="2.806544567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:59.806262736 +0000 UTC m=+3030.694179825" watchObservedRunningTime="2026-10-30 19:28:59.806544567 +0000 UTC m=+3030.694461656"
Oct 30 19:29:00.699061 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:00.699027    4267 generic.go:356] "Generic (PLEG): container finished" podID="621a91fe-7351-42c9-842f-a22388611929" containerID="5e97569acdf7a67809e2ba38c6bb336986e70d1ca13e4225a9c5c43bd0094e06" exitCode=0
Oct 30 19:29:00.699456 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:00.699069    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8" event={"ID":"621a91fe-7351-42c9-842f-a22388611929","Type":"ContainerDied","Data":"5e97569acdf7a67809e2ba38c6bb336986e70d1ca13e4225a9c5c43bd0094e06"}
Oct 30 19:29:01.833853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:01.833825    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:29:01.972692 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:01.972587    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621a91fe-7351-42c9-842f-a22388611929-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621a91fe-7351-42c9-842f-a22388611929-maas-api-openshift-ca\") pod \"621a91fe-7351-42c9-842f-a22388611929\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") "
Oct 30 19:29:01.972874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:01.972721    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621a91fe-7351-42c9-842f-a22388611929-kube-api-access-rvzzx\" (UniqueName: \"kubernetes.io/projected/621a91fe-7351-42c9-842f-a22388611929-kube-api-access-rvzzx\") pod \"621a91fe-7351-42c9-842f-a22388611929\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") "
Oct 30 19:29:01.972874 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:01.972797    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621a91fe-7351-42c9-842f-a22388611929-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621a91fe-7351-42c9-842f-a22388611929-maas-api-xks-ca\") pod \"621a91fe-7351-42c9-842f-a22388611929\" (UID: \"621a91fe-7351-42c9-842f-a22388611929\") "
Oct 30 19:29:01.973068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:01.973029    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621a91fe-7351-42c9-842f-a22388611929-maas-api-openshift-ca" pod "621a91fe-7351-42c9-842f-a22388611929" (UID: "621a91fe-7351-42c9-842f-a22388611929"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:01.973205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:01.973184    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621a91fe-7351-42c9-842f-a22388611929-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:29:01.975667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:01.975639    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621a91fe-7351-42c9-842f-a22388611929-kube-api-access-rvzzx" pod "621a91fe-7351-42c9-842f-a22388611929" (UID: "621a91fe-7351-42c9-842f-a22388611929"). InnerVolumeSpecName "kube-api-access-rvzzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:01.975826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:01.975807    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621a91fe-7351-42c9-842f-a22388611929-maas-api-xks-ca" pod "621a91fe-7351-42c9-842f-a22388611929" (UID: "621a91fe-7351-42c9-842f-a22388611929"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:02.074721 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:02.074679    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvzzx\" (UniqueName: \"kubernetes.io/projected/621a91fe-7351-42c9-842f-a22388611929-kube-api-access-rvzzx\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:29:02.074721 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:02.074713    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621a91fe-7351-42c9-842f-a22388611929-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:29:02.711803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:02.711774    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"
Oct 30 19:29:02.711803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:02.711793    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8" event={"ID":"621a91fe-7351-42c9-842f-a22388611929","Type":"ContainerDied","Data":"5d5842f8a277a9a8197f26dbe9d500191f7bbbad424ba04b626b99a4d87f7bf6"}
Oct 30 19:29:02.712024 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:02.711838    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d5842f8a277a9a8197f26dbe9d500191f7bbbad424ba04b626b99a4d87f7bf6"
Oct 30 19:29:20.158036 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:29:20.157991    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:29:23.189391 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.189348    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"]
Oct 30 19:29:23.189940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.189836    4267 cpu_manager.go:414] "containerMap: removing container" podUID="621a91fe-7351-42c9-842f-a22388611929" containerName="revoke-keys"
Oct 30 19:29:23.189940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.189860    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621a91fe-7351-42c9-842f-a22388611929" containerName="revoke-keys"
Oct 30 19:29:23.190068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.189979    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="621a91fe-7351-42c9-842f-a22388611929" containerName="revoke-keys"
Oct 30 19:29:23.193564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.193538    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:29:23.249926 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.249890    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"]
Oct 30 19:29:23.271873 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.271835    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmwr\" (UniqueName: \"kubernetes.io/projected/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-kube-api-access-8lmwr\") pod \"payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp\" (UID: \"d50f1502-7584-4d92-9a7d-8a0a5de4af5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:29:23.272059 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.271969    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp\" (UID: \"d50f1502-7584-4d92-9a7d-8a0a5de4af5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:29:23.296246 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.296217    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-da238b\""
Oct 30 19:29:23.296401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.296329    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5cf2d8f7-dockercfg-6hbvd\""
Oct 30 19:29:23.356269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.356242    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"]
Oct 30 19:29:23.359727 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.359710    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:29:23.373446 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.373425    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmwr\" (UniqueName: \"kubernetes.io/projected/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-kube-api-access-8lmwr\") pod \"payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp\" (UID: \"d50f1502-7584-4d92-9a7d-8a0a5de4af5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:29:23.373574 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.373480    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp\" (UID: \"d50f1502-7584-4d92-9a7d-8a0a5de4af5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:29:23.374087 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.374068    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp\" (UID: \"d50f1502-7584-4d92-9a7d-8a0a5de4af5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:29:23.417784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.417749    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"]
Oct 30 19:29:23.430399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.430370    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmwr\" (UniqueName: \"kubernetes.io/projected/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-kube-api-access-8lmwr\") pod \"payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp\" (UID: \"d50f1502-7584-4d92-9a7d-8a0a5de4af5b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:29:23.474389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.474283    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbnkp\" (UniqueName: \"kubernetes.io/projected/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-kube-api-access-wbnkp\") pod \"payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf\" (UID: \"24ab18d4-125f-4ead-9815-0b6fe82e0f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:29:23.474565 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.474395    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf\" (UID: \"24ab18d4-125f-4ead-9815-0b6fe82e0f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:29:23.509055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.509020    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:29:23.575430 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.575276    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf\" (UID: \"24ab18d4-125f-4ead-9815-0b6fe82e0f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:29:23.575828 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.575677    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbnkp\" (UniqueName: \"kubernetes.io/projected/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-kube-api-access-wbnkp\") pod \"payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf\" (UID: \"24ab18d4-125f-4ead-9815-0b6fe82e0f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:29:23.576240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.576189    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf\" (UID: \"24ab18d4-125f-4ead-9815-0b6fe82e0f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:29:23.633559 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.633529    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbnkp\" (UniqueName: \"kubernetes.io/projected/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-kube-api-access-wbnkp\") pod \"payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf\" (UID: \"24ab18d4-125f-4ead-9815-0b6fe82e0f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:29:23.672609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.672580    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:29:23.882270 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:23.882246    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"]
Oct 30 19:29:23.883677 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:29:23.883649    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50f1502_7584_4d92_9a7d_8a0a5de4af5b.slice/crio-b92bd866e4647a0d5ab5c7fb6b57d80c3397f83e3e510c8db41c3773fecd51ff WatchSource:0}: Error finding container b92bd866e4647a0d5ab5c7fb6b57d80c3397f83e3e510c8db41c3773fecd51ff: Status 404 returned error can't find the container with id b92bd866e4647a0d5ab5c7fb6b57d80c3397f83e3e510c8db41c3773fecd51ff
Oct 30 19:29:24.045164 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:24.045132    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"]
Oct 30 19:29:24.046548 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:29:24.046481    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ab18d4_125f_4ead_9815_0b6fe82e0f0d.slice/crio-cb8ec174c7751935aba22c6d42d832fbcda9b285c49fe9919ac3237ae68d6ccd WatchSource:0}: Error finding container cb8ec174c7751935aba22c6d42d832fbcda9b285c49fe9919ac3237ae68d6ccd: Status 404 returned error can't find the container with id cb8ec174c7751935aba22c6d42d832fbcda9b285c49fe9919ac3237ae68d6ccd
Oct 30 19:29:24.842135 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:24.842096    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf" event={"ID":"24ab18d4-125f-4ead-9815-0b6fe82e0f0d","Type":"ContainerStarted","Data":"4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119"}
Oct 30 19:29:24.842667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:24.842157    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf" event={"ID":"24ab18d4-125f-4ead-9815-0b6fe82e0f0d","Type":"ContainerStarted","Data":"cb8ec174c7751935aba22c6d42d832fbcda9b285c49fe9919ac3237ae68d6ccd"}
Oct 30 19:29:24.842667 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:24.842301    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:29:24.843728 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:24.843707    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp" event={"ID":"d50f1502-7584-4d92-9a7d-8a0a5de4af5b","Type":"ContainerStarted","Data":"8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c"}
Oct 30 19:29:24.843845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:24.843742    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp" event={"ID":"d50f1502-7584-4d92-9a7d-8a0a5de4af5b","Type":"ContainerStarted","Data":"b92bd866e4647a0d5ab5c7fb6b57d80c3397f83e3e510c8db41c3773fecd51ff"}
Oct 30 19:29:24.843845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:24.843795    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:29:24.958865 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:24.958802    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf" podStartSLOduration=1.9587886270000001 podStartE2EDuration="1.958788627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:24.956030113 +0000 UTC m=+3055.843947202" watchObservedRunningTime="2026-10-30 19:29:24.958788627 +0000 UTC m=+3055.846705734"
Oct 30 19:29:25.071104 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:25.071039    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp" podStartSLOduration=2.071017581 podStartE2EDuration="2.071017581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:25.070087635 +0000 UTC m=+3055.958004748" watchObservedRunningTime="2026-10-30 19:29:25.071017581 +0000 UTC m=+3055.958934672"
Oct 30 19:29:32.740134 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:32.740092    4267 scope.go:122] "RemoveContainer" containerID="587064e8e147c0182c0b8f8a5432eb29646d765864220b85f35e72065d756e0a"
Oct 30 19:29:35.854261 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:35.854234    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:29:35.854696 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:35.854306    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:29:48.057748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.057715    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"]
Oct 30 19:29:48.062319 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.062301    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.114693 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.114657    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkwhd\" (UniqueName: \"kubernetes.io/projected/3572f883-d574-4f76-b740-24728811e272-kube-api-access-dkwhd\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.114864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.114720    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3572f883-d574-4f76-b740-24728811e272-tls-certs\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.114864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.114822    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-dshm\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.114864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.114853    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-tmp-dir\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.114977 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.114875    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-model-cache\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.114977 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.114929    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-home\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.159393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.159358    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"]
Oct 30 19:29:48.167216 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.167189    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-da238b\"/\"openshift-service-ca.crt\""
Oct 30 19:29:48.167372 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.167355    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-da238b\"/\"simulator-single-main-da238b-kserve-self-signed-certs\""
Oct 30 19:29:48.167467 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.167451    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-da238b\"/\"default-dockercfg-dd4g8\""
Oct 30 19:29:48.167523 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.167467    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-da238b\"/\"kube-root-ca.crt\""
Oct 30 19:29:48.215359 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.215322    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-dshm\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.215359 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.215366    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-tmp-dir\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.215690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.215389    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-model-cache\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.215690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.215408    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-home\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.215690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.215438    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwhd\" (UniqueName: \"kubernetes.io/projected/3572f883-d574-4f76-b740-24728811e272-kube-api-access-dkwhd\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.215690 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.215520    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3572f883-d574-4f76-b740-24728811e272-tls-certs\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.215926 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.215858    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-tmp-dir\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.215926 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.215899    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-home\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.216007 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.215919    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-model-cache\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.217863 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.217839    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-dshm\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.218125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.218108    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3572f883-d574-4f76-b740-24728811e272-tls-certs\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.277435 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.277404    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkwhd\" (UniqueName: \"kubernetes.io/projected/3572f883-d574-4f76-b740-24728811e272-kube-api-access-dkwhd\") pod \"simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") " pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.375724 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.375638    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:48.752662 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.752631    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"]
Oct 30 19:29:48.753680 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:29:48.753652    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3572f883_d574_4f76_b740_24728811e272.slice/crio-dc8e080a8744b4261f5a2508eab6fbd63bf18e82d76534ab8cef13be17bac780 WatchSource:0}: Error finding container dc8e080a8744b4261f5a2508eab6fbd63bf18e82d76534ab8cef13be17bac780: Status 404 returned error can't find the container with id dc8e080a8744b4261f5a2508eab6fbd63bf18e82d76534ab8cef13be17bac780
Oct 30 19:29:48.977673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.977632    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb" event={"ID":"3572f883-d574-4f76-b740-24728811e272","Type":"ContainerStarted","Data":"8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc"}
Oct 30 19:29:48.977883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:48.977682    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb" event={"ID":"3572f883-d574-4f76-b740-24728811e272","Type":"ContainerStarted","Data":"dc8e080a8744b4261f5a2508eab6fbd63bf18e82d76534ab8cef13be17bac780"}
Oct 30 19:29:49.096902 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:49.096802    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb" podStartSLOduration=1.096787275 podStartE2EDuration="1.096787275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:49.09418267 +0000 UTC m=+3079.982099755" watchObservedRunningTime="2026-10-30 19:29:49.096787275 +0000 UTC m=+3079.984704365"
Oct 30 19:29:49.376791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:49.376705    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:50.046928 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:50.046897    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"]
Oct 30 19:29:50.097288 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:50.097253    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fmr7"]
Oct 30 19:29:50.158915 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:29:50.158875    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:29:51.219392 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:51.219357    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9b3fa9-6c20-4a41-a2a0-37544c5f944c" path="/var/lib/kubelet/pods/af9b3fa9-6c20-4a41-a2a0-37544c5f944c/volumes"
Oct 30 19:29:58.376008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:58.375970    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:58.389342 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:58.389311    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:29:59.046836 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:29:59.046804    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:30:00.164551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.164514    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"]
Oct 30 19:30:00.169331 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.169311    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"]
Oct 30 19:30:00.169462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.169433    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"
Oct 30 19:30:00.173694 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.173670    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"
Oct 30 19:30:00.213871 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.213844    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9741079-ff69-43e9-910f-3c2eb656c8cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-khw8g\" (UID: \"b9741079-ff69-43e9-910f-3c2eb656c8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"
Oct 30 19:30:00.214052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.213881    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnz8p\" (UniqueName: \"kubernetes.io/projected/b9741079-ff69-43e9-910f-3c2eb656c8cf-kube-api-access-qnz8p\") pod \"maas-api-key-cleanup-29889810-khw8g\" (UID: \"b9741079-ff69-43e9-910f-3c2eb656c8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"
Oct 30 19:30:00.214052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.213908    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbb88\" (UniqueName: \"kubernetes.io/projected/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-kube-api-access-sbb88\") pod \"maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5\" (UID: \"794e7cc3-3a7f-4216-ae0e-26b339e0af5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"
Oct 30 19:30:00.214052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.214039    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5\" (UID: \"794e7cc3-3a7f-4216-ae0e-26b339e0af5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"
Oct 30 19:30:00.224248 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.224221    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:30:00.224432 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.224262    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:30:00.228174 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.228143    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"]
Oct 30 19:30:00.274100 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.274068    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:30:00.278408 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.278379    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"]
Oct 30 19:30:00.315128 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.315096    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9741079-ff69-43e9-910f-3c2eb656c8cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-khw8g\" (UID: \"b9741079-ff69-43e9-910f-3c2eb656c8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"
Oct 30 19:30:00.315328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.315143    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnz8p\" (UniqueName: \"kubernetes.io/projected/b9741079-ff69-43e9-910f-3c2eb656c8cf-kube-api-access-qnz8p\") pod \"maas-api-key-cleanup-29889810-khw8g\" (UID: \"b9741079-ff69-43e9-910f-3c2eb656c8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"
Oct 30 19:30:00.315328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.315169    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbb88\" (UniqueName: \"kubernetes.io/projected/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-kube-api-access-sbb88\") pod \"maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5\" (UID: \"794e7cc3-3a7f-4216-ae0e-26b339e0af5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"
Oct 30 19:30:00.315328 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.315200    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5\" (UID: \"794e7cc3-3a7f-4216-ae0e-26b339e0af5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"
Oct 30 19:30:00.315912 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.315893    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9741079-ff69-43e9-910f-3c2eb656c8cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-khw8g\" (UID: \"b9741079-ff69-43e9-910f-3c2eb656c8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"
Oct 30 19:30:00.316079 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.316054    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5\" (UID: \"794e7cc3-3a7f-4216-ae0e-26b339e0af5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"
Oct 30 19:30:00.372790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.372758    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnz8p\" (UniqueName: \"kubernetes.io/projected/b9741079-ff69-43e9-910f-3c2eb656c8cf-kube-api-access-qnz8p\") pod \"maas-api-key-cleanup-29889810-khw8g\" (UID: \"b9741079-ff69-43e9-910f-3c2eb656c8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"
Oct 30 19:30:00.372945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.372876    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbb88\" (UniqueName: \"kubernetes.io/projected/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-kube-api-access-sbb88\") pod \"maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5\" (UID: \"794e7cc3-3a7f-4216-ae0e-26b339e0af5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"
Oct 30 19:30:00.484697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.484667    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"
Oct 30 19:30:00.490399 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.490376    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"
Oct 30 19:30:00.865242 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.865208    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"]
Oct 30 19:30:00.865975 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:30:00.865946    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794e7cc3_3a7f_4216_ae0e_26b339e0af5a.slice/crio-a8cfbf52a258c0e80cfad8080140c56ef1b68bba244c77bc91f75b558de43dee WatchSource:0}: Error finding container a8cfbf52a258c0e80cfad8080140c56ef1b68bba244c77bc91f75b558de43dee: Status 404 returned error can't find the container with id a8cfbf52a258c0e80cfad8080140c56ef1b68bba244c77bc91f75b558de43dee
Oct 30 19:30:00.913201 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:00.913171    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"]
Oct 30 19:30:00.914627 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:30:00.914605    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9741079_ff69_43e9_910f_3c2eb656c8cf.slice/crio-4f6247f26b6ea95c5b814ee24d50af7dcd42443877db08764357fab4df5575a6 WatchSource:0}: Error finding container 4f6247f26b6ea95c5b814ee24d50af7dcd42443877db08764357fab4df5575a6: Status 404 returned error can't find the container with id 4f6247f26b6ea95c5b814ee24d50af7dcd42443877db08764357fab4df5575a6
Oct 30 19:30:01.048229 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:01.048184    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g" event={"ID":"b9741079-ff69-43e9-910f-3c2eb656c8cf","Type":"ContainerStarted","Data":"4f6247f26b6ea95c5b814ee24d50af7dcd42443877db08764357fab4df5575a6"}
Oct 30 19:30:01.049538 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:01.049487    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5" event={"ID":"794e7cc3-3a7f-4216-ae0e-26b339e0af5a","Type":"ContainerStarted","Data":"a8cfbf52a258c0e80cfad8080140c56ef1b68bba244c77bc91f75b558de43dee"}
Oct 30 19:30:02.057848 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:02.057812    4267 generic.go:356] "Generic (PLEG): container finished" podID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerID="d623b4c6e1d51b2262b792ca1f0b0cda0745b1174826157b69e8cd08ed8b7a89" exitCode=60
Oct 30 19:30:02.058330 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:02.057900    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g" event={"ID":"b9741079-ff69-43e9-910f-3c2eb656c8cf","Type":"ContainerDied","Data":"d623b4c6e1d51b2262b792ca1f0b0cda0745b1174826157b69e8cd08ed8b7a89"}
Oct 30 19:30:02.058330 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:02.058156    4267 scope.go:122] "RemoveContainer" containerID="d623b4c6e1d51b2262b792ca1f0b0cda0745b1174826157b69e8cd08ed8b7a89"
Oct 30 19:30:02.059753 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:02.059733    4267 generic.go:356] "Generic (PLEG): container finished" podID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerID="f62b526bea37873a7a53b98ae0300941d210d4c3e9e8a40f00df91959337c8a0" exitCode=60
Oct 30 19:30:02.059863 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:02.059837    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5" event={"ID":"794e7cc3-3a7f-4216-ae0e-26b339e0af5a","Type":"ContainerDied","Data":"f62b526bea37873a7a53b98ae0300941d210d4c3e9e8a40f00df91959337c8a0"}
Oct 30 19:30:02.059997 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:02.059984    4267 scope.go:122] "RemoveContainer" containerID="f62b526bea37873a7a53b98ae0300941d210d4c3e9e8a40f00df91959337c8a0"
Oct 30 19:30:03.066591 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:03.066504    4267 generic.go:356] "Generic (PLEG): container finished" podID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerID="f5c59bdcf2ac9bf9a0df6ee490f2d9cb66be3f534e6edf52f2a55c434a0262bf" exitCode=60
Oct 30 19:30:03.067049 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:03.066584    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g" event={"ID":"b9741079-ff69-43e9-910f-3c2eb656c8cf","Type":"ContainerDied","Data":"f5c59bdcf2ac9bf9a0df6ee490f2d9cb66be3f534e6edf52f2a55c434a0262bf"}
Oct 30 19:30:03.067049 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:03.066674    4267 scope.go:122] "RemoveContainer" containerID="d623b4c6e1d51b2262b792ca1f0b0cda0745b1174826157b69e8cd08ed8b7a89"
Oct 30 19:30:03.067049 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:03.066908    4267 scope.go:122] "RemoveContainer" containerID="f5c59bdcf2ac9bf9a0df6ee490f2d9cb66be3f534e6edf52f2a55c434a0262bf"
Oct 30 19:30:03.067196 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:30:03.067177    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-khw8g_odh-ai-gateway-infra(b9741079-ff69-43e9-910f-3c2eb656c8cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf"
Oct 30 19:30:03.068548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:03.068512    4267 generic.go:356] "Generic (PLEG): container finished" podID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerID="3b206eb02a4361550b6dc3e6ccee3bba74c5229b93385b380aca8d7d51b6192d" exitCode=60
Oct 30 19:30:03.068679 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:03.068543    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5" event={"ID":"794e7cc3-3a7f-4216-ae0e-26b339e0af5a","Type":"ContainerDied","Data":"3b206eb02a4361550b6dc3e6ccee3bba74c5229b93385b380aca8d7d51b6192d"}
Oct 30 19:30:03.068771 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:03.068757    4267 scope.go:122] "RemoveContainer" containerID="3b206eb02a4361550b6dc3e6ccee3bba74c5229b93385b380aca8d7d51b6192d"
Oct 30 19:30:03.068966 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:30:03.068950    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5_odh-ai-gateway-infra(794e7cc3-3a7f-4216-ae0e-26b339e0af5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a"
Oct 30 19:30:03.078343 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:03.078320    4267 scope.go:122] "RemoveContainer" containerID="f62b526bea37873a7a53b98ae0300941d210d4c3e9e8a40f00df91959337c8a0"
Oct 30 19:30:04.076502 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:04.076472    4267 scope.go:122] "RemoveContainer" containerID="3b206eb02a4361550b6dc3e6ccee3bba74c5229b93385b380aca8d7d51b6192d"
Oct 30 19:30:04.077103 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:30:04.076783    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5_odh-ai-gateway-infra(794e7cc3-3a7f-4216-ae0e-26b339e0af5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a"
Oct 30 19:30:04.078233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:04.078213    4267 scope.go:122] "RemoveContainer" containerID="f5c59bdcf2ac9bf9a0df6ee490f2d9cb66be3f534e6edf52f2a55c434a0262bf"
Oct 30 19:30:04.078422 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:30:04.078405    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-khw8g_odh-ai-gateway-infra(b9741079-ff69-43e9-910f-3c2eb656c8cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf"
Oct 30 19:30:16.214149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:16.214052    4267 scope.go:122] "RemoveContainer" containerID="f5c59bdcf2ac9bf9a0df6ee490f2d9cb66be3f534e6edf52f2a55c434a0262bf"
Oct 30 19:30:16.214671 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:16.214292    4267 scope.go:122] "RemoveContainer" containerID="3b206eb02a4361550b6dc3e6ccee3bba74c5229b93385b380aca8d7d51b6192d"
Oct 30 19:30:17.157366 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:17.157321    4267 generic.go:356] "Generic (PLEG): container finished" podID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerID="9455ef990865423aea30ba51f0b95bdfc9d0269b61144660ce93fc8a6878e567" exitCode=60
Oct 30 19:30:17.157637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:17.157550    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5" event={"ID":"794e7cc3-3a7f-4216-ae0e-26b339e0af5a","Type":"ContainerDied","Data":"9455ef990865423aea30ba51f0b95bdfc9d0269b61144660ce93fc8a6878e567"}
Oct 30 19:30:17.157637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:17.157615    4267 scope.go:122] "RemoveContainer" containerID="3b206eb02a4361550b6dc3e6ccee3bba74c5229b93385b380aca8d7d51b6192d"
Oct 30 19:30:17.158161 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:17.158091    4267 scope.go:122] "RemoveContainer" containerID="9455ef990865423aea30ba51f0b95bdfc9d0269b61144660ce93fc8a6878e567"
Oct 30 19:30:17.158528 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:30:17.158455    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5_odh-ai-gateway-infra(794e7cc3-3a7f-4216-ae0e-26b339e0af5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a"
Oct 30 19:30:17.164008 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:17.163979    4267 generic.go:356] "Generic (PLEG): container finished" podID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerID="875d211dddc49f1fd0598c040a79a62bcf58fd533f90010edc66b14cf479a1b4" exitCode=60
Oct 30 19:30:17.164132 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:17.164049    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g" event={"ID":"b9741079-ff69-43e9-910f-3c2eb656c8cf","Type":"ContainerDied","Data":"875d211dddc49f1fd0598c040a79a62bcf58fd533f90010edc66b14cf479a1b4"}
Oct 30 19:30:17.164465 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:17.164448    4267 scope.go:122] "RemoveContainer" containerID="875d211dddc49f1fd0598c040a79a62bcf58fd533f90010edc66b14cf479a1b4"
Oct 30 19:30:17.164758 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:30:17.164724    4267 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-khw8g_odh-ai-gateway-infra(b9741079-ff69-43e9-910f-3c2eb656c8cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf"
Oct 30 19:30:17.173784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:17.173761    4267 scope.go:122] "RemoveContainer" containerID="f5c59bdcf2ac9bf9a0df6ee490f2d9cb66be3f534e6edf52f2a55c434a0262bf"
Oct 30 19:30:18.286395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.286358    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"]
Oct 30 19:30:18.399809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.399771    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"]
Oct 30 19:30:18.431149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.431116    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"
Oct 30 19:30:18.500685 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.500650    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-maas-api-service-ca\") pod \"794e7cc3-3a7f-4216-ae0e-26b339e0af5a\" (UID: \"794e7cc3-3a7f-4216-ae0e-26b339e0af5a\") "
Oct 30 19:30:18.500884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.500717    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-kube-api-access-sbb88\" (UniqueName: \"kubernetes.io/projected/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-kube-api-access-sbb88\") pod \"794e7cc3-3a7f-4216-ae0e-26b339e0af5a\" (UID: \"794e7cc3-3a7f-4216-ae0e-26b339e0af5a\") "
Oct 30 19:30:18.501022 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.500999    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-maas-api-service-ca" pod "794e7cc3-3a7f-4216-ae0e-26b339e0af5a" (UID: "794e7cc3-3a7f-4216-ae0e-26b339e0af5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:18.502845 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.502820    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-kube-api-access-sbb88" pod "794e7cc3-3a7f-4216-ae0e-26b339e0af5a" (UID: "794e7cc3-3a7f-4216-ae0e-26b339e0af5a"). InnerVolumeSpecName "kube-api-access-sbb88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:18.541791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.541736    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"
Oct 30 19:30:18.601850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.601815    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:18.601850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.601846    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbb88\" (UniqueName: \"kubernetes.io/projected/794e7cc3-3a7f-4216-ae0e-26b339e0af5a-kube-api-access-sbb88\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:18.702710 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.702680    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9741079-ff69-43e9-910f-3c2eb656c8cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9741079-ff69-43e9-910f-3c2eb656c8cf-maas-api-service-ca\") pod \"b9741079-ff69-43e9-910f-3c2eb656c8cf\" (UID: \"b9741079-ff69-43e9-910f-3c2eb656c8cf\") "
Oct 30 19:30:18.702925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.702742    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9741079-ff69-43e9-910f-3c2eb656c8cf-kube-api-access-qnz8p\" (UniqueName: \"kubernetes.io/projected/b9741079-ff69-43e9-910f-3c2eb656c8cf-kube-api-access-qnz8p\") pod \"b9741079-ff69-43e9-910f-3c2eb656c8cf\" (UID: \"b9741079-ff69-43e9-910f-3c2eb656c8cf\") "
Oct 30 19:30:18.703070 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.703045    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9741079-ff69-43e9-910f-3c2eb656c8cf-maas-api-service-ca" pod "b9741079-ff69-43e9-910f-3c2eb656c8cf" (UID: "b9741079-ff69-43e9-910f-3c2eb656c8cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:18.704896 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.704876    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9741079-ff69-43e9-910f-3c2eb656c8cf-kube-api-access-qnz8p" pod "b9741079-ff69-43e9-910f-3c2eb656c8cf" (UID: "b9741079-ff69-43e9-910f-3c2eb656c8cf"). InnerVolumeSpecName "kube-api-access-qnz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:18.803524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.803422    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnz8p\" (UniqueName: \"kubernetes.io/projected/b9741079-ff69-43e9-910f-3c2eb656c8cf-kube-api-access-qnz8p\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:18.803524 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:18.803459    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9741079-ff69-43e9-910f-3c2eb656c8cf-maas-api-service-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:19.047000 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.046964    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"]
Oct 30 19:30:19.049465 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.049436    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-286lk"]
Oct 30 19:30:19.182962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.182871    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"
Oct 30 19:30:19.182962 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.182900    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5" event={"ID":"794e7cc3-3a7f-4216-ae0e-26b339e0af5a","Type":"ContainerDied","Data":"a8cfbf52a258c0e80cfad8080140c56ef1b68bba244c77bc91f75b558de43dee"}
Oct 30 19:30:19.183214 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.182966    4267 scope.go:122] "RemoveContainer" containerID="9455ef990865423aea30ba51f0b95bdfc9d0269b61144660ce93fc8a6878e567"
Oct 30 19:30:19.185054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.185025    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g" event={"ID":"b9741079-ff69-43e9-910f-3c2eb656c8cf","Type":"ContainerDied","Data":"4f6247f26b6ea95c5b814ee24d50af7dcd42443877db08764357fab4df5575a6"}
Oct 30 19:30:19.185190 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.185120    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"
Oct 30 19:30:19.194579 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.194555    4267 scope.go:122] "RemoveContainer" containerID="875d211dddc49f1fd0598c040a79a62bcf58fd533f90010edc66b14cf479a1b4"
Oct 30 19:30:19.219682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.219653    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f92b12a-193f-44bb-afc7-1f9501266f1c" path="/var/lib/kubelet/pods/4f92b12a-193f-44bb-afc7-1f9501266f1c/volumes"
Oct 30 19:30:19.351585 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.351550    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"]
Oct 30 19:30:19.355909 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.355879    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-da238b-2988981frlj5"]
Oct 30 19:30:19.466316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.466287    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"]
Oct 30 19:30:19.470764 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:19.470740    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-khw8g"]
Oct 30 19:30:20.157248 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:30:20.157210    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:30:21.220185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:21.220149    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" path="/var/lib/kubelet/pods/794e7cc3-3a7f-4216-ae0e-26b339e0af5a/volumes"
Oct 30 19:30:21.220786 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:21.220585    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf" path="/var/lib/kubelet/pods/b9741079-ff69-43e9-910f-3c2eb656c8cf/volumes"
Oct 30 19:30:24.327581 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.327544    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"]
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.327928    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.327939    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.327953    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.327958    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.327977    4267 cpu_manager.go:414] "containerMap: removing container" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.327982    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.327992    4267 cpu_manager.go:414] "containerMap: removing container" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.327997    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.328051    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.328061    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.328070    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.328076    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerName="cleanup"
Oct 30 19:30:24.328096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.328081    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerName="cleanup"
Oct 30 19:30:24.332398 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.332374    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:24.390841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.390601    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:30:24.390841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.390672    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:30:24.393241 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.393212    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"]
Oct 30 19:30:24.439452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.439425    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:30:24.439602 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.439462    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:30:24.458717 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.458689    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c496f6-jgc42\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:24.458844 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.458719    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c496f6-jgc42\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:24.458844 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.458747    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh8dt\" (UniqueName: \"kubernetes.io/projected/37282a31-72c0-451e-b2cb-bde8b50c6c95-kube-api-access-gh8dt\") pod \"maas-api-revoke-sub-86c496f6-jgc42\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:24.560147 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.560115    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh8dt\" (UniqueName: \"kubernetes.io/projected/37282a31-72c0-451e-b2cb-bde8b50c6c95-kube-api-access-gh8dt\") pod \"maas-api-revoke-sub-86c496f6-jgc42\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:24.560314 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.560259    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c496f6-jgc42\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:24.560314 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.560287    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c496f6-jgc42\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:24.560966 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.560943    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c496f6-jgc42\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:24.562632 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.562611    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c496f6-jgc42\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:24.619174 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.619117    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh8dt\" (UniqueName: \"kubernetes.io/projected/37282a31-72c0-451e-b2cb-bde8b50c6c95-kube-api-access-gh8dt\") pod \"maas-api-revoke-sub-86c496f6-jgc42\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:24.645948 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:24.645924    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:25.015733 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:25.015704    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"]
Oct 30 19:30:25.016914 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:30:25.016886    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37282a31_72c0_451e_b2cb_bde8b50c6c95.slice/crio-cbf86830b0de50a0d522c63fb1fd0a0dcbf05cef41e89f17726a6e8a8a69084d WatchSource:0}: Error finding container cbf86830b0de50a0d522c63fb1fd0a0dcbf05cef41e89f17726a6e8a8a69084d: Status 404 returned error can't find the container with id cbf86830b0de50a0d522c63fb1fd0a0dcbf05cef41e89f17726a6e8a8a69084d
Oct 30 19:30:25.221209 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:25.221180    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42" event={"ID":"37282a31-72c0-451e-b2cb-bde8b50c6c95","Type":"ContainerStarted","Data":"cbf86830b0de50a0d522c63fb1fd0a0dcbf05cef41e89f17726a6e8a8a69084d"}
Oct 30 19:30:26.228047 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:26.228007    4267 generic.go:356] "Generic (PLEG): container finished" podID="37282a31-72c0-451e-b2cb-bde8b50c6c95" containerID="d5b73dd2348275634bdd6517c270aefe73f901d541a04771349310e1ed3af6fe" exitCode=0
Oct 30 19:30:26.228457 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:26.228088    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42" event={"ID":"37282a31-72c0-451e-b2cb-bde8b50c6c95","Type":"ContainerDied","Data":"d5b73dd2348275634bdd6517c270aefe73f901d541a04771349310e1ed3af6fe"}
Oct 30 19:30:27.371082 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:27.371053    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:27.488838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:27.488753    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-xks-ca\") pod \"37282a31-72c0-451e-b2cb-bde8b50c6c95\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") "
Oct 30 19:30:27.488838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:27.488814    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-openshift-ca\") pod \"37282a31-72c0-451e-b2cb-bde8b50c6c95\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") "
Oct 30 19:30:27.489041 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:27.488952    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37282a31-72c0-451e-b2cb-bde8b50c6c95-kube-api-access-gh8dt\" (UniqueName: \"kubernetes.io/projected/37282a31-72c0-451e-b2cb-bde8b50c6c95-kube-api-access-gh8dt\") pod \"37282a31-72c0-451e-b2cb-bde8b50c6c95\" (UID: \"37282a31-72c0-451e-b2cb-bde8b50c6c95\") "
Oct 30 19:30:27.489217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:27.489190    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-openshift-ca" pod "37282a31-72c0-451e-b2cb-bde8b50c6c95" (UID: "37282a31-72c0-451e-b2cb-bde8b50c6c95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:27.490994 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:27.490969    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37282a31-72c0-451e-b2cb-bde8b50c6c95-kube-api-access-gh8dt" pod "37282a31-72c0-451e-b2cb-bde8b50c6c95" (UID: "37282a31-72c0-451e-b2cb-bde8b50c6c95"). InnerVolumeSpecName "kube-api-access-gh8dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:27.491073 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:27.491000    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-xks-ca" pod "37282a31-72c0-451e-b2cb-bde8b50c6c95" (UID: "37282a31-72c0-451e-b2cb-bde8b50c6c95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:27.590395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:27.590343    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:27.590395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:27.590389    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh8dt\" (UniqueName: \"kubernetes.io/projected/37282a31-72c0-451e-b2cb-bde8b50c6c95-kube-api-access-gh8dt\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:27.590395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:27.590400    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37282a31-72c0-451e-b2cb-bde8b50c6c95-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:28.239862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:28.239831    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"
Oct 30 19:30:28.239862 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:28.239845    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42" event={"ID":"37282a31-72c0-451e-b2cb-bde8b50c6c95","Type":"ContainerDied","Data":"cbf86830b0de50a0d522c63fb1fd0a0dcbf05cef41e89f17726a6e8a8a69084d"}
Oct 30 19:30:28.240089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:28.239888    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf86830b0de50a0d522c63fb1fd0a0dcbf05cef41e89f17726a6e8a8a69084d"
Oct 30 19:30:31.048527 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:31.048471    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"]
Oct 30 19:30:31.097878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:31.097842    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-qw9q9"]
Oct 30 19:30:31.219034 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:31.219001    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f84332a-be22-4aaf-8506-8780fe73d1d9" path="/var/lib/kubelet/pods/6f84332a-be22-4aaf-8506-8780fe73d1d9/volumes"
Oct 30 19:30:32.767162 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:32.767131    4267 scope.go:122] "RemoveContainer" containerID="27168df8906d3b8741c368c8c065fb7f6114ef89a15a9018ce7c426373b602ca"
Oct 30 19:30:36.422637 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.422597    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"]
Oct 30 19:30:36.423121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.422960    4267 cpu_manager.go:414] "containerMap: removing container" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerName="cleanup"
Oct 30 19:30:36.423121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.422971    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794e7cc3-3a7f-4216-ae0e-26b339e0af5a" containerName="cleanup"
Oct 30 19:30:36.423121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.422991    4267 cpu_manager.go:414] "containerMap: removing container" podUID="37282a31-72c0-451e-b2cb-bde8b50c6c95" containerName="revoke-keys"
Oct 30 19:30:36.423121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.422996    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37282a31-72c0-451e-b2cb-bde8b50c6c95" containerName="revoke-keys"
Oct 30 19:30:36.423121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.423012    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerName="cleanup"
Oct 30 19:30:36.423121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.423017    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerName="cleanup"
Oct 30 19:30:36.423121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.423082    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="37282a31-72c0-451e-b2cb-bde8b50c6c95" containerName="revoke-keys"
Oct 30 19:30:36.423121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.423092    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9741079-ff69-43e9-910f-3c2eb656c8cf" containerName="cleanup"
Oct 30 19:30:36.427552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.427531    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:36.476641 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.476602    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:36.476841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.476648    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:36.476841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.476764    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxklg\" (UniqueName: \"kubernetes.io/projected/75dbbfac-c407-447c-ba14-39424d1c4889-kube-api-access-nxklg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:36.480275 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.480250    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:30:36.529877 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.529843    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:30:36.530090 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.530070    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:30:36.530228 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.530213    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:30:36.532999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.532979    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"]
Oct 30 19:30:36.577821 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.577785    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:36.578016 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.577904    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxklg\" (UniqueName: \"kubernetes.io/projected/75dbbfac-c407-447c-ba14-39424d1c4889-kube-api-access-nxklg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:36.578016 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.577988    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:36.578743 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.578712    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:36.580650 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.580625    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:36.636416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.636376    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxklg\" (UniqueName: \"kubernetes.io/projected/75dbbfac-c407-447c-ba14-39424d1c4889-kube-api-access-nxklg\") pod \"maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:36.740552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:36.740502    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:37.119827 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:37.119802    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"]
Oct 30 19:30:37.121944 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:30:37.121915    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75dbbfac_c407_447c_ba14_39424d1c4889.slice/crio-2f47ba3ec8643ef3652d58bc274d7d3729c0f3f26f6769c6c367bc0813518a97 WatchSource:0}: Error finding container 2f47ba3ec8643ef3652d58bc274d7d3729c0f3f26f6769c6c367bc0813518a97: Status 404 returned error can't find the container with id 2f47ba3ec8643ef3652d58bc274d7d3729c0f3f26f6769c6c367bc0813518a97
Oct 30 19:30:37.289369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:37.289336    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9" event={"ID":"75dbbfac-c407-447c-ba14-39424d1c4889","Type":"ContainerStarted","Data":"2f47ba3ec8643ef3652d58bc274d7d3729c0f3f26f6769c6c367bc0813518a97"}
Oct 30 19:30:38.296787 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:38.296702    4267 generic.go:356] "Generic (PLEG): container finished" podID="75dbbfac-c407-447c-ba14-39424d1c4889" containerID="d3e41e5cfb31d6197892bfc8fefc40a90ab3ca94be2beb2fc28d13a0cc9c0d9b" exitCode=0
Oct 30 19:30:38.296787 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:38.296750    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9" event={"ID":"75dbbfac-c407-447c-ba14-39424d1c4889","Type":"ContainerDied","Data":"d3e41e5cfb31d6197892bfc8fefc40a90ab3ca94be2beb2fc28d13a0cc9c0d9b"}
Oct 30 19:30:39.451670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:39.451642    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:39.503652 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:39.503613    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75dbbfac-c407-447c-ba14-39424d1c4889-kube-api-access-nxklg\" (UniqueName: \"kubernetes.io/projected/75dbbfac-c407-447c-ba14-39424d1c4889-kube-api-access-nxklg\") pod \"75dbbfac-c407-447c-ba14-39424d1c4889\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") "
Oct 30 19:30:39.503847 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:39.503739    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-xks-ca\") pod \"75dbbfac-c407-447c-ba14-39424d1c4889\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") "
Oct 30 19:30:39.503847 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:39.503762    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-openshift-ca\") pod \"75dbbfac-c407-447c-ba14-39424d1c4889\" (UID: \"75dbbfac-c407-447c-ba14-39424d1c4889\") "
Oct 30 19:30:39.504191 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:39.504157    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-openshift-ca" pod "75dbbfac-c407-447c-ba14-39424d1c4889" (UID: "75dbbfac-c407-447c-ba14-39424d1c4889"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:39.505875 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:39.505848    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-xks-ca" pod "75dbbfac-c407-447c-ba14-39424d1c4889" (UID: "75dbbfac-c407-447c-ba14-39424d1c4889"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:39.506019 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:39.505995    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75dbbfac-c407-447c-ba14-39424d1c4889-kube-api-access-nxklg" pod "75dbbfac-c407-447c-ba14-39424d1c4889" (UID: "75dbbfac-c407-447c-ba14-39424d1c4889"). InnerVolumeSpecName "kube-api-access-nxklg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:39.604811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:39.604720    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:39.604811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:39.604754    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75dbbfac-c407-447c-ba14-39424d1c4889-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:39.604811 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:39.604764    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxklg\" (UniqueName: \"kubernetes.io/projected/75dbbfac-c407-447c-ba14-39424d1c4889-kube-api-access-nxklg\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:40.309188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:40.309146    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9" event={"ID":"75dbbfac-c407-447c-ba14-39424d1c4889","Type":"ContainerDied","Data":"2f47ba3ec8643ef3652d58bc274d7d3729c0f3f26f6769c6c367bc0813518a97"}
Oct 30 19:30:40.309188 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:40.309168    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"
Oct 30 19:30:40.309410 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:40.309205    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f47ba3ec8643ef3652d58bc274d7d3729c0f3f26f6769c6c367bc0813518a97"
Oct 30 19:30:41.094958 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:41.094921    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"]
Oct 30 19:30:47.846582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:47.846540    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"]
Oct 30 19:30:47.952066 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:47.952025    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"]
Oct 30 19:30:50.157656 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:30:50.157621    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:30:52.847525 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:52.847460    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp" podUID="d50f1502-7584-4d92-9a7d-8a0a5de4af5b" containerName="payload-processing" containerID="cri-o://8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c" gracePeriod=25
Oct 30 19:30:52.952825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:52.952769    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf" podUID="24ab18d4-125f-4ead-9815-0b6fe82e0f0d" containerName="payload-pre-processing" containerID="cri-o://4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119" gracePeriod=25
Oct 30 19:30:53.456276 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:53.456240    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"]
Oct 30 19:30:53.505347 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:53.505305    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-da238b-xqpp9"]
Oct 30 19:30:55.219437 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:55.219394    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75dbbfac-c407-447c-ba14-39424d1c4889" path="/var/lib/kubelet/pods/75dbbfac-c407-447c-ba14-39424d1c4889/volumes"
Oct 30 19:30:55.852539 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:55.852482    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp" podUID="d50f1502-7584-4d92-9a7d-8a0a5de4af5b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:30:55.852706 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:55.852482    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf" podUID="24ab18d4-125f-4ead-9815-0b6fe82e0f0d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:30:56.114826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.114739    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb" podUID="3572f883-d574-4f76-b740-24728811e272" containerName="main" containerID="cri-o://8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc" gracePeriod=45
Oct 30 19:30:56.383280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.383244    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:30:56.402062 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.402026    4267 generic.go:356] "Generic (PLEG): container finished" podID="3572f883-d574-4f76-b740-24728811e272" containerID="8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc" exitCode=0
Oct 30 19:30:56.402260 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.402063    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb" event={"ID":"3572f883-d574-4f76-b740-24728811e272","Type":"ContainerDied","Data":"8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc"}
Oct 30 19:30:56.402260 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.402108    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"
Oct 30 19:30:56.402260 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.402123    4267 scope.go:122] "RemoveContainer" containerID="8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc"
Oct 30 19:30:56.402260 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.402112    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb" event={"ID":"3572f883-d574-4f76-b740-24728811e272","Type":"ContainerDied","Data":"dc8e080a8744b4261f5a2508eab6fbd63bf18e82d76534ab8cef13be17bac780"}
Oct 30 19:30:56.411419 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.411406    4267 scope.go:122] "RemoveContainer" containerID="8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc"
Oct 30 19:30:56.411735 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:30:56.411714    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc\": container with ID starting with 8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc not found: ID does not exist" containerID="8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc"
Oct 30 19:30:56.411859 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.411742    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc"} err="failed to get container status \"8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc\": rpc error: code = NotFound desc = could not find container \"8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc\": container with ID starting with 8856b5799cc59a7eeedc8236cf9e620bd8f1d13ec67a563e6ada767bac57ecfc not found: ID does not exist"
Oct 30 19:30:56.469901 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.469866    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-model-cache\") pod \"3572f883-d574-4f76-b740-24728811e272\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") "
Oct 30 19:30:56.470111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.469908    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3572f883-d574-4f76-b740-24728811e272-tls-certs\" (UniqueName: \"kubernetes.io/secret/3572f883-d574-4f76-b740-24728811e272-tls-certs\") pod \"3572f883-d574-4f76-b740-24728811e272\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") "
Oct 30 19:30:56.470111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.469949    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-tmp-dir\") pod \"3572f883-d574-4f76-b740-24728811e272\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") "
Oct 30 19:30:56.470111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.469977    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-dshm\") pod \"3572f883-d574-4f76-b740-24728811e272\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") "
Oct 30 19:30:56.470111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.469994    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-home\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-home\") pod \"3572f883-d574-4f76-b740-24728811e272\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") "
Oct 30 19:30:56.470111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.470092    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3572f883-d574-4f76-b740-24728811e272-kube-api-access-dkwhd\" (UniqueName: \"kubernetes.io/projected/3572f883-d574-4f76-b740-24728811e272-kube-api-access-dkwhd\") pod \"3572f883-d574-4f76-b740-24728811e272\" (UID: \"3572f883-d574-4f76-b740-24728811e272\") "
Oct 30 19:30:56.470426 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.470215    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-model-cache" pod "3572f883-d574-4f76-b740-24728811e272" (UID: "3572f883-d574-4f76-b740-24728811e272"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.470426 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.470253    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-tmp-dir" pod "3572f883-d574-4f76-b740-24728811e272" (UID: "3572f883-d574-4f76-b740-24728811e272"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.470426 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.470358    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-home" pod "3572f883-d574-4f76-b740-24728811e272" (UID: "3572f883-d574-4f76-b740-24728811e272"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.470614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.470540    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.470614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.470584    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.470614 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.470599    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.472400 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.472375    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3572f883-d574-4f76-b740-24728811e272-tls-certs" pod "3572f883-d574-4f76-b740-24728811e272" (UID: "3572f883-d574-4f76-b740-24728811e272"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:56.472531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.472414    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-dshm" pod "3572f883-d574-4f76-b740-24728811e272" (UID: "3572f883-d574-4f76-b740-24728811e272"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:56.472531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.472458    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3572f883-d574-4f76-b740-24728811e272-kube-api-access-dkwhd" pod "3572f883-d574-4f76-b740-24728811e272" (UID: "3572f883-d574-4f76-b740-24728811e272"). InnerVolumeSpecName "kube-api-access-dkwhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:56.571701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.571659    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3572f883-d574-4f76-b740-24728811e272-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.571701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.571694    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3572f883-d574-4f76-b740-24728811e272-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.571701 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.571709    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkwhd\" (UniqueName: \"kubernetes.io/projected/3572f883-d574-4f76-b740-24728811e272-kube-api-access-dkwhd\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:56.823267 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.823231    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"]
Oct 30 19:30:56.827081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:56.827058    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-da238b/simulator-single-main-da238b-kserve-65fd7d66d8-gxvdb"]
Oct 30 19:30:57.219205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:30:57.219173    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3572f883-d574-4f76-b740-24728811e272" path="/var/lib/kubelet/pods/3572f883-d574-4f76-b740-24728811e272/volumes"
Oct 30 19:31:02.098128 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.098093    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"]
Oct 30 19:31:02.147611 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.147578    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f84m8"]
Oct 30 19:31:02.220285 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.220243    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"]
Oct 30 19:31:02.220654 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.220642    4267 cpu_manager.go:414] "containerMap: removing container" podUID="75dbbfac-c407-447c-ba14-39424d1c4889" containerName="revoke-keys"
Oct 30 19:31:02.220725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.220655    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75dbbfac-c407-447c-ba14-39424d1c4889" containerName="revoke-keys"
Oct 30 19:31:02.220725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.220680    4267 cpu_manager.go:414] "containerMap: removing container" podUID="3572f883-d574-4f76-b740-24728811e272" containerName="main"
Oct 30 19:31:02.220725 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.220685    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3572f883-d574-4f76-b740-24728811e272" containerName="main"
Oct 30 19:31:02.220822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.220748    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="75dbbfac-c407-447c-ba14-39424d1c4889" containerName="revoke-keys"
Oct 30 19:31:02.220822 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.220760    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="3572f883-d574-4f76-b740-24728811e272" containerName="main"
Oct 30 19:31:02.223676 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.223660    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.279233 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.279197    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:31:02.279426 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.279235    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:31:02.279486 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.279460    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:31:02.279563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.279483    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:31:02.282476 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.282448    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"]
Oct 30 19:31:02.332111 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.332073    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg4hj\" (UniqueName: \"kubernetes.io/projected/60938420-e80e-45ac-870e-3b81bc3f55e9-kube-api-access-wg4hj\") pod \"maas-api-revoke-sub-3d4156f1-xwx92\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.332298 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.332137    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx92\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.332298 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.332254    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx92\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.433323 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.433227    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx92\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.433323 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.433268    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx92\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.433618 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.433344    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg4hj\" (UniqueName: \"kubernetes.io/projected/60938420-e80e-45ac-870e-3b81bc3f55e9-kube-api-access-wg4hj\") pod \"maas-api-revoke-sub-3d4156f1-xwx92\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.434015 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.433994    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx92\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.435847 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.435821    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xwx92\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.491115 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.491070    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg4hj\" (UniqueName: \"kubernetes.io/projected/60938420-e80e-45ac-870e-3b81bc3f55e9-kube-api-access-wg4hj\") pod \"maas-api-revoke-sub-3d4156f1-xwx92\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.536788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.536747    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:02.911027 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:02.910997    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"]
Oct 30 19:31:02.912019 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:31:02.911989    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60938420_e80e_45ac_870e_3b81bc3f55e9.slice/crio-6d1751b46a63f05420f462f1788c5365d40e9c239a4ad25a49c741cc950d8163 WatchSource:0}: Error finding container 6d1751b46a63f05420f462f1788c5365d40e9c239a4ad25a49c741cc950d8163: Status 404 returned error can't find the container with id 6d1751b46a63f05420f462f1788c5365d40e9c239a4ad25a49c741cc950d8163
Oct 30 19:31:03.220064 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:03.220034    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621a91fe-7351-42c9-842f-a22388611929" path="/var/lib/kubelet/pods/621a91fe-7351-42c9-842f-a22388611929/volumes"
Oct 30 19:31:03.446073 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:03.446037    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92" event={"ID":"60938420-e80e-45ac-870e-3b81bc3f55e9","Type":"ContainerStarted","Data":"26a798a61df6da662cb2a76476927d38c8b1ee8a7d3f9df50c68276923aa0682"}
Oct 30 19:31:03.446272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:03.446087    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92" event={"ID":"60938420-e80e-45ac-870e-3b81bc3f55e9","Type":"ContainerStarted","Data":"6d1751b46a63f05420f462f1788c5365d40e9c239a4ad25a49c741cc950d8163"}
Oct 30 19:31:03.561577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:03.561466    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92" podStartSLOduration=1.561450652 podStartE2EDuration="1.561450652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:03.559180403 +0000 UTC m=+3154.447097492" watchObservedRunningTime="2026-10-30 19:31:03.561450652 +0000 UTC m=+3154.449367740"
Oct 30 19:31:04.455100 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:04.455059    4267 generic.go:356] "Generic (PLEG): container finished" podID="60938420-e80e-45ac-870e-3b81bc3f55e9" containerID="26a798a61df6da662cb2a76476927d38c8b1ee8a7d3f9df50c68276923aa0682" exitCode=0
Oct 30 19:31:04.455548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:04.455137    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92" event={"ID":"60938420-e80e-45ac-870e-3b81bc3f55e9","Type":"ContainerDied","Data":"26a798a61df6da662cb2a76476927d38c8b1ee8a7d3f9df50c68276923aa0682"}
Oct 30 19:31:05.589244 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.589221    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:05.665734 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.665708    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-openshift-ca\") pod \"60938420-e80e-45ac-870e-3b81bc3f55e9\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") "
Oct 30 19:31:05.665872 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.665762    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-xks-ca\") pod \"60938420-e80e-45ac-870e-3b81bc3f55e9\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") "
Oct 30 19:31:05.665872 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.665859    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60938420-e80e-45ac-870e-3b81bc3f55e9-kube-api-access-wg4hj\" (UniqueName: \"kubernetes.io/projected/60938420-e80e-45ac-870e-3b81bc3f55e9-kube-api-access-wg4hj\") pod \"60938420-e80e-45ac-870e-3b81bc3f55e9\" (UID: \"60938420-e80e-45ac-870e-3b81bc3f55e9\") "
Oct 30 19:31:05.666093 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.666068    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-openshift-ca" pod "60938420-e80e-45ac-870e-3b81bc3f55e9" (UID: "60938420-e80e-45ac-870e-3b81bc3f55e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:05.667784 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.667765    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-xks-ca" pod "60938420-e80e-45ac-870e-3b81bc3f55e9" (UID: "60938420-e80e-45ac-870e-3b81bc3f55e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:05.667859 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.667841    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60938420-e80e-45ac-870e-3b81bc3f55e9-kube-api-access-wg4hj" pod "60938420-e80e-45ac-870e-3b81bc3f55e9" (UID: "60938420-e80e-45ac-870e-3b81bc3f55e9"). InnerVolumeSpecName "kube-api-access-wg4hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:05.766823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.766742    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg4hj\" (UniqueName: \"kubernetes.io/projected/60938420-e80e-45ac-870e-3b81bc3f55e9-kube-api-access-wg4hj\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.766823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.766769    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.766823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.766779    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60938420-e80e-45ac-870e-3b81bc3f55e9-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.852125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.852082    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf" podUID="24ab18d4-125f-4ead-9815-0b6fe82e0f0d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:31:05.852320 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:05.852082    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp" podUID="d50f1502-7584-4d92-9a7d-8a0a5de4af5b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:31:06.047081 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:06.046983    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"]
Oct 30 19:31:06.050308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:06.050281    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dhgj9"]
Oct 30 19:31:06.467118 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:06.467089    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"
Oct 30 19:31:06.467318 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:06.467126    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92" event={"ID":"60938420-e80e-45ac-870e-3b81bc3f55e9","Type":"ContainerDied","Data":"6d1751b46a63f05420f462f1788c5365d40e9c239a4ad25a49c741cc950d8163"}
Oct 30 19:31:06.467318 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:06.467199    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d1751b46a63f05420f462f1788c5365d40e9c239a4ad25a49c741cc950d8163"
Oct 30 19:31:07.220592 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:07.220558    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99e607b-6d95-4df7-bd23-3f1ab968b1c5" path="/var/lib/kubelet/pods/e99e607b-6d95-4df7-bd23-3f1ab968b1c5/volumes"
Oct 30 19:31:13.102500 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.102420    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:31:13.198658 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.198634    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:31:13.237426 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.237396    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-plugins-config-volume\") pod \"d50f1502-7584-4d92-9a7d-8a0a5de4af5b\" (UID: \"d50f1502-7584-4d92-9a7d-8a0a5de4af5b\") "
Oct 30 19:31:13.237628 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.237527    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-kube-api-access-8lmwr\" (UniqueName: \"kubernetes.io/projected/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-kube-api-access-8lmwr\") pod \"d50f1502-7584-4d92-9a7d-8a0a5de4af5b\" (UID: \"d50f1502-7584-4d92-9a7d-8a0a5de4af5b\") "
Oct 30 19:31:13.237819 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.237793    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-plugins-config-volume" pod "d50f1502-7584-4d92-9a7d-8a0a5de4af5b" (UID: "d50f1502-7584-4d92-9a7d-8a0a5de4af5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:13.239791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.239766    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-kube-api-access-8lmwr" pod "d50f1502-7584-4d92-9a7d-8a0a5de4af5b" (UID: "d50f1502-7584-4d92-9a7d-8a0a5de4af5b"). InnerVolumeSpecName "kube-api-access-8lmwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:13.339067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.339034    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-plugins-config-volume\") pod \"24ab18d4-125f-4ead-9815-0b6fe82e0f0d\" (UID: \"24ab18d4-125f-4ead-9815-0b6fe82e0f0d\") "
Oct 30 19:31:13.339253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.339174    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-kube-api-access-wbnkp\" (UniqueName: \"kubernetes.io/projected/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-kube-api-access-wbnkp\") pod \"24ab18d4-125f-4ead-9815-0b6fe82e0f0d\" (UID: \"24ab18d4-125f-4ead-9815-0b6fe82e0f0d\") "
Oct 30 19:31:13.339628 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.339484    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-plugins-config-volume" pod "24ab18d4-125f-4ead-9815-0b6fe82e0f0d" (UID: "24ab18d4-125f-4ead-9815-0b6fe82e0f0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:13.339628 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.339527    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:13.339628 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.339550    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmwr\" (UniqueName: \"kubernetes.io/projected/d50f1502-7584-4d92-9a7d-8a0a5de4af5b-kube-api-access-8lmwr\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:13.341976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.341952    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-kube-api-access-wbnkp" pod "24ab18d4-125f-4ead-9815-0b6fe82e0f0d" (UID: "24ab18d4-125f-4ead-9815-0b6fe82e0f0d"). InnerVolumeSpecName "kube-api-access-wbnkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:13.440322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.440281    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:13.440322 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.440317    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbnkp\" (UniqueName: \"kubernetes.io/projected/24ab18d4-125f-4ead-9815-0b6fe82e0f0d-kube-api-access-wbnkp\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:13.507118 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.507084    4267 generic.go:356] "Generic (PLEG): container finished" podID="24ab18d4-125f-4ead-9815-0b6fe82e0f0d" containerID="4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119" exitCode=0
Oct 30 19:31:13.507313 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.507134    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"
Oct 30 19:31:13.507313 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.507184    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf" event={"ID":"24ab18d4-125f-4ead-9815-0b6fe82e0f0d","Type":"ContainerDied","Data":"4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119"}
Oct 30 19:31:13.507313 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.507243    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf" event={"ID":"24ab18d4-125f-4ead-9815-0b6fe82e0f0d","Type":"ContainerDied","Data":"cb8ec174c7751935aba22c6d42d832fbcda9b285c49fe9919ac3237ae68d6ccd"}
Oct 30 19:31:13.507313 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.507274    4267 scope.go:122] "RemoveContainer" containerID="4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119"
Oct 30 19:31:13.508695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.508668    4267 generic.go:356] "Generic (PLEG): container finished" podID="d50f1502-7584-4d92-9a7d-8a0a5de4af5b" containerID="8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c" exitCode=0
Oct 30 19:31:13.508804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.508710    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"
Oct 30 19:31:13.508804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.508707    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp" event={"ID":"d50f1502-7584-4d92-9a7d-8a0a5de4af5b","Type":"ContainerDied","Data":"8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c"}
Oct 30 19:31:13.509200 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.508829    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp" event={"ID":"d50f1502-7584-4d92-9a7d-8a0a5de4af5b","Type":"ContainerDied","Data":"b92bd866e4647a0d5ab5c7fb6b57d80c3397f83e3e510c8db41c3773fecd51ff"}
Oct 30 19:31:13.517599 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.517582    4267 scope.go:122] "RemoveContainer" containerID="4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119"
Oct 30 19:31:13.517895 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:31:13.517877    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119\": container with ID starting with 4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119 not found: ID does not exist" containerID="4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119"
Oct 30 19:31:13.517945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.517904    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119"} err="failed to get container status \"4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119\": rpc error: code = NotFound desc = could not find container \"4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119\": container with ID starting with 4f850103353a1bebc67d6cff3a8c01050ff3ad7b2d300f94134564fafe539119 not found: ID does not exist"
Oct 30 19:31:13.517945 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.517920    4267 scope.go:122] "RemoveContainer" containerID="8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c"
Oct 30 19:31:13.526395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.526375    4267 scope.go:122] "RemoveContainer" containerID="8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c"
Oct 30 19:31:13.526648 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:31:13.526628    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c\": container with ID starting with 8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c not found: ID does not exist" containerID="8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c"
Oct 30 19:31:13.526709 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.526656    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c"} err="failed to get container status \"8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c\": rpc error: code = NotFound desc = could not find container \"8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c\": container with ID starting with 8ef99bc0c4055e0faba8c266dc4ae3c282c4f8e4c01a6a45beff8b5ef8d4134c not found: ID does not exist"
Oct 30 19:31:13.672342 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.672302    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"]
Oct 30 19:31:13.674628 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.674599    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-da238b-9645d4c59pkgkp"]
Oct 30 19:31:13.784555 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.784454    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"]
Oct 30 19:31:13.788485 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:13.788444    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-da238b-685b9trvxf"]
Oct 30 19:31:15.218556 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:15.218522    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ab18d4-125f-4ead-9815-0b6fe82e0f0d" path="/var/lib/kubelet/pods/24ab18d4-125f-4ead-9815-0b6fe82e0f0d/volumes"
Oct 30 19:31:15.218978 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:15.218916    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50f1502-7584-4d92-9a7d-8a0a5de4af5b" path="/var/lib/kubelet/pods/d50f1502-7584-4d92-9a7d-8a0a5de4af5b/volumes"
Oct 30 19:31:17.048550 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:17.048511    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"]
Oct 30 19:31:17.098238 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:17.098205    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9rhbz"]
Oct 30 19:31:17.218475 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:17.218439    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2fba27-0b4b-4047-bbc2-b5539b32a13a" path="/var/lib/kubelet/pods/fa2fba27-0b4b-4047-bbc2-b5539b32a13a/volumes"
Oct 30 19:31:20.157841 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:31:20.157799    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:31:21.333395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.333358    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"]
Oct 30 19:31:21.333859 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.333588    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh" podUID="3c757ca6-2bb6-4274-8ee3-40a16a01ca29" containerName="manager" containerID="cri-o://142bdc818342cab7170b0bf6829a6c8ff61aa0c4a1658b26e1b7bcf6dad5ec79" gracePeriod=10
Oct 30 19:31:21.558521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.558464    4267 generic.go:356] "Generic (PLEG): container finished" podID="3c757ca6-2bb6-4274-8ee3-40a16a01ca29" containerID="142bdc818342cab7170b0bf6829a6c8ff61aa0c4a1658b26e1b7bcf6dad5ec79" exitCode=0
Oct 30 19:31:21.558691 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.558540    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh" event={"ID":"3c757ca6-2bb6-4274-8ee3-40a16a01ca29","Type":"ContainerDied","Data":"142bdc818342cab7170b0bf6829a6c8ff61aa0c4a1658b26e1b7bcf6dad5ec79"}
Oct 30 19:31:21.584266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.584212    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 19:31:21.599352 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.599328    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-extensions-socket-volume\") pod \"3c757ca6-2bb6-4274-8ee3-40a16a01ca29\" (UID: \"3c757ca6-2bb6-4274-8ee3-40a16a01ca29\") "
Oct 30 19:31:21.599472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.599370    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-kube-api-access-dpffl\" (UniqueName: \"kubernetes.io/projected/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-kube-api-access-dpffl\") pod \"3c757ca6-2bb6-4274-8ee3-40a16a01ca29\" (UID: \"3c757ca6-2bb6-4274-8ee3-40a16a01ca29\") "
Oct 30 19:31:21.599707 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.599687    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-extensions-socket-volume" pod "3c757ca6-2bb6-4274-8ee3-40a16a01ca29" (UID: "3c757ca6-2bb6-4274-8ee3-40a16a01ca29"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:21.601315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.601296    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-kube-api-access-dpffl" pod "3c757ca6-2bb6-4274-8ee3-40a16a01ca29" (UID: "3c757ca6-2bb6-4274-8ee3-40a16a01ca29"). InnerVolumeSpecName "kube-api-access-dpffl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:21.700227 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.700187    4267 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-extensions-socket-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:21.700227 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:21.700221    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpffl\" (UniqueName: \"kubernetes.io/projected/3c757ca6-2bb6-4274-8ee3-40a16a01ca29-kube-api-access-dpffl\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:22.565597 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:22.565560    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"
Oct 30 19:31:22.565597 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:22.565590    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh" event={"ID":"3c757ca6-2bb6-4274-8ee3-40a16a01ca29","Type":"ContainerDied","Data":"68343a645a690d88810d3f00acce33830d0769a31e365e95e27b6ec9dc767bb1"}
Oct 30 19:31:22.566174 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:22.565649    4267 scope.go:122] "RemoveContainer" containerID="142bdc818342cab7170b0bf6829a6c8ff61aa0c4a1658b26e1b7bcf6dad5ec79"
Oct 30 19:31:22.781036 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:22.780995    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"]
Oct 30 19:31:22.831293 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:22.831201    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-b4gdh"]
Oct 30 19:31:23.218825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:23.218786    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c757ca6-2bb6-4274-8ee3-40a16a01ca29" path="/var/lib/kubelet/pods/3c757ca6-2bb6-4274-8ee3-40a16a01ca29/volumes"
Oct 30 19:31:32.797864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:32.797832    4267 scope.go:122] "RemoveContainer" containerID="c866e72959eeb86c34cde89e3ba4042b2c959ebcc059d6d959506b7dac186bd6"
Oct 30 19:31:32.807887 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:32.807860    4267 scope.go:122] "RemoveContainer" containerID="828b00de9c5517f714f2ffebcfd79a9ea9f5b460426deb6c5410f96579bb0fe4"
Oct 30 19:31:32.816067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:32.816048    4267 scope.go:122] "RemoveContainer" containerID="a0af4543acca5cb92e3333758f09107f565af8d452411a0ff73eb8a16ff0d769"
Oct 30 19:31:50.157684 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:31:50.157598    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:31:51.047963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:51.047920    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"]
Oct 30 19:31:51.097781 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:51.097742    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-9sm69"]
Oct 30 19:31:51.219390 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:31:51.219359    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92449a5-20b6-444b-aa39-a69c029a4eac" path="/var/lib/kubelet/pods/c92449a5-20b6-444b-aa39-a69c029a4eac/volumes"
Oct 30 19:32:20.064681 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:20.064640    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"]
Oct 30 19:32:20.118997 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:20.116511    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gtflg"]
Oct 30 19:32:20.157413 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:32:20.157368    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:32:21.218884 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:21.218851    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f8604f-6f86-4e10-9bef-405bea72e7e4" path="/var/lib/kubelet/pods/84f8604f-6f86-4e10-9bef-405bea72e7e4/volumes"
Oct 30 19:32:24.959185 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959138    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"]
Oct 30 19:32:24.959748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959673    4267 cpu_manager.go:414] "containerMap: removing container" podUID="24ab18d4-125f-4ead-9815-0b6fe82e0f0d" containerName="payload-pre-processing"
Oct 30 19:32:24.959748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959697    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ab18d4-125f-4ead-9815-0b6fe82e0f0d" containerName="payload-pre-processing"
Oct 30 19:32:24.959748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959723    4267 cpu_manager.go:414] "containerMap: removing container" podUID="60938420-e80e-45ac-870e-3b81bc3f55e9" containerName="revoke-keys"
Oct 30 19:32:24.959748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959731    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60938420-e80e-45ac-870e-3b81bc3f55e9" containerName="revoke-keys"
Oct 30 19:32:24.959984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959761    4267 cpu_manager.go:414] "containerMap: removing container" podUID="d50f1502-7584-4d92-9a7d-8a0a5de4af5b" containerName="payload-processing"
Oct 30 19:32:24.959984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959768    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50f1502-7584-4d92-9a7d-8a0a5de4af5b" containerName="payload-processing"
Oct 30 19:32:24.959984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959779    4267 cpu_manager.go:414] "containerMap: removing container" podUID="3c757ca6-2bb6-4274-8ee3-40a16a01ca29" containerName="manager"
Oct 30 19:32:24.959984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959788    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c757ca6-2bb6-4274-8ee3-40a16a01ca29" containerName="manager"
Oct 30 19:32:24.959984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959891    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ab18d4-125f-4ead-9815-0b6fe82e0f0d" containerName="payload-pre-processing"
Oct 30 19:32:24.959984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959909    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="60938420-e80e-45ac-870e-3b81bc3f55e9" containerName="revoke-keys"
Oct 30 19:32:24.959984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959921    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50f1502-7584-4d92-9a7d-8a0a5de4af5b" containerName="payload-processing"
Oct 30 19:32:24.959984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.959929    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c757ca6-2bb6-4274-8ee3-40a16a01ca29" containerName="manager"
Oct 30 19:32:24.963320 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:24.963297    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"
Oct 30 19:32:25.009082 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.009052    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"]
Oct 30 19:32:25.056675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.056634    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l677f\""
Oct 30 19:32:25.056863 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.056795    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g498v\" (UniqueName: \"kubernetes.io/projected/28f82707-5133-4bd8-b36a-62042290ac08-kube-api-access-g498v\") pod \"kuadrant-operator-controller-manager-57bb9bfff-8wbf4\" (UID: \"28f82707-5133-4bd8-b36a-62042290ac08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"
Oct 30 19:32:25.056863 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.056843    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28f82707-5133-4bd8-b36a-62042290ac08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-8wbf4\" (UID: \"28f82707-5133-4bd8-b36a-62042290ac08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"
Oct 30 19:32:25.157791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.157759    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g498v\" (UniqueName: \"kubernetes.io/projected/28f82707-5133-4bd8-b36a-62042290ac08-kube-api-access-g498v\") pod \"kuadrant-operator-controller-manager-57bb9bfff-8wbf4\" (UID: \"28f82707-5133-4bd8-b36a-62042290ac08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"
Oct 30 19:32:25.157988 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.157796    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28f82707-5133-4bd8-b36a-62042290ac08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-8wbf4\" (UID: \"28f82707-5133-4bd8-b36a-62042290ac08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"
Oct 30 19:32:25.158227 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.158209    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28f82707-5133-4bd8-b36a-62042290ac08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-8wbf4\" (UID: \"28f82707-5133-4bd8-b36a-62042290ac08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"
Oct 30 19:32:25.216363 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.216282    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g498v\" (UniqueName: \"kubernetes.io/projected/28f82707-5133-4bd8-b36a-62042290ac08-kube-api-access-g498v\") pod \"kuadrant-operator-controller-manager-57bb9bfff-8wbf4\" (UID: \"28f82707-5133-4bd8-b36a-62042290ac08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"
Oct 30 19:32:25.275441 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.275408    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"
Oct 30 19:32:25.711106 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.711075    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"]
Oct 30 19:32:25.712823 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:32:25.712794    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f82707_5133_4bd8_b36a_62042290ac08.slice/crio-7f3f7d39257af9aeddc8bcf77438fa4e354c8e2b3e561a48a84e7ba0a03322f9 WatchSource:0}: Error finding container 7f3f7d39257af9aeddc8bcf77438fa4e354c8e2b3e561a48a84e7ba0a03322f9: Status 404 returned error can't find the container with id 7f3f7d39257af9aeddc8bcf77438fa4e354c8e2b3e561a48a84e7ba0a03322f9
Oct 30 19:32:25.897970 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.897926    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4" event={"ID":"28f82707-5133-4bd8-b36a-62042290ac08","Type":"ContainerStarted","Data":"ad6f996e11abd5267683001375416f87be03f1fce96aedb166d241952303eade"}
Oct 30 19:32:25.898172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.898141    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"
Oct 30 19:32:25.898238 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:25.898182    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4" event={"ID":"28f82707-5133-4bd8-b36a-62042290ac08","Type":"ContainerStarted","Data":"7f3f7d39257af9aeddc8bcf77438fa4e354c8e2b3e561a48a84e7ba0a03322f9"}
Oct 30 19:32:26.019554 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:26.019428    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4" podStartSLOduration=2.019413316 podStartE2EDuration="2.019413316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:26.017246473 +0000 UTC m=+3236.905163553" watchObservedRunningTime="2026-10-30 19:32:26.019413316 +0000 UTC m=+3236.907330405"
Oct 30 19:32:32.859338 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:32.859305    4267 scope.go:122] "RemoveContainer" containerID="a642752498c492edc04029d8155c6aaf5d36d476c8460779613497b5a532f4ef"
Oct 30 19:32:32.868638 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:32.868620    4267 scope.go:122] "RemoveContainer" containerID="5300b95f8a1cd1569a302635d306c84a1a0f8807036abfb8ea22a5ae3f1e1b93"
Oct 30 19:32:36.906655 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:36.906619    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-8wbf4"
Oct 30 19:32:50.068517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:50.067462    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"]
Oct 30 19:32:50.099516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:50.099431    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6q4b9"]
Oct 30 19:32:50.364336 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:32:50.364237    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:32:51.219509 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.219463    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e8b7ce-7df7-4a36-adf8-42994c032319" path="/var/lib/kubelet/pods/78e8b7ce-7df7-4a36-adf8-42994c032319/volumes"
Oct 30 19:32:51.812968 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.812935    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"]
Oct 30 19:32:51.820881 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.820849    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:51.923482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.923419    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:32:51.923482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.923457    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:32:51.923482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.923455    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:32:51.923790 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.923586    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:32:51.926150 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.926128    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"]
Oct 30 19:32:51.957218 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.957181    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p4zv7\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:51.957389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.957347    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffzbd\" (UniqueName: \"kubernetes.io/projected/8e91d125-96d0-4a51-acb8-bd6060ff2766-kube-api-access-ffzbd\") pod \"maas-api-revoke-sub-d576e68b-p4zv7\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:51.957465 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:51.957391    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p4zv7\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:52.058602 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:52.058550    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffzbd\" (UniqueName: \"kubernetes.io/projected/8e91d125-96d0-4a51-acb8-bd6060ff2766-kube-api-access-ffzbd\") pod \"maas-api-revoke-sub-d576e68b-p4zv7\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:52.058820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:52.058620    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p4zv7\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:52.058820 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:52.058688    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p4zv7\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:52.059305 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:52.059277    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p4zv7\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:52.061141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:52.061120    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p4zv7\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:52.116205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:52.116127    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffzbd\" (UniqueName: \"kubernetes.io/projected/8e91d125-96d0-4a51-acb8-bd6060ff2766-kube-api-access-ffzbd\") pod \"maas-api-revoke-sub-d576e68b-p4zv7\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:52.134479 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:52.134449    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:52.511978 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:52.511946    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"]
Oct 30 19:32:52.513127 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:32:52.513099    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e91d125_96d0_4a51_acb8_bd6060ff2766.slice/crio-c17f6d07d95d2eeea5e91986f36d283e8e7ce8e2857ca404c48a92cafc2b3b81 WatchSource:0}: Error finding container c17f6d07d95d2eeea5e91986f36d283e8e7ce8e2857ca404c48a92cafc2b3b81: Status 404 returned error can't find the container with id c17f6d07d95d2eeea5e91986f36d283e8e7ce8e2857ca404c48a92cafc2b3b81
Oct 30 19:32:53.076702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:53.076599    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7" event={"ID":"8e91d125-96d0-4a51-acb8-bd6060ff2766","Type":"ContainerStarted","Data":"5177b3f29e818b0e1e4db1b3b8026f45ce21ce7f21d15b0c81cd01e690ff6e25"}
Oct 30 19:32:53.076702 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:53.076666    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7" event={"ID":"8e91d125-96d0-4a51-acb8-bd6060ff2766","Type":"ContainerStarted","Data":"c17f6d07d95d2eeea5e91986f36d283e8e7ce8e2857ca404c48a92cafc2b3b81"}
Oct 30 19:32:53.191353 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:53.191295    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7" podStartSLOduration=2.191276581 podStartE2EDuration="2.191276581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:53.190795455 +0000 UTC m=+3264.078712544" watchObservedRunningTime="2026-10-30 19:32:53.191276581 +0000 UTC m=+3264.079193670"
Oct 30 19:32:54.082950 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:54.082912    4267 generic.go:356] "Generic (PLEG): container finished" podID="8e91d125-96d0-4a51-acb8-bd6060ff2766" containerID="5177b3f29e818b0e1e4db1b3b8026f45ce21ce7f21d15b0c81cd01e690ff6e25" exitCode=0
Oct 30 19:32:54.083427 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:54.082998    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7" event={"ID":"8e91d125-96d0-4a51-acb8-bd6060ff2766","Type":"ContainerDied","Data":"5177b3f29e818b0e1e4db1b3b8026f45ce21ce7f21d15b0c81cd01e690ff6e25"}
Oct 30 19:32:55.220447 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:55.220422    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:32:55.287339 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:55.287304    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-xks-ca\") pod \"8e91d125-96d0-4a51-acb8-bd6060ff2766\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") "
Oct 30 19:32:55.287544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:55.287432    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e91d125-96d0-4a51-acb8-bd6060ff2766-kube-api-access-ffzbd\" (UniqueName: \"kubernetes.io/projected/8e91d125-96d0-4a51-acb8-bd6060ff2766-kube-api-access-ffzbd\") pod \"8e91d125-96d0-4a51-acb8-bd6060ff2766\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") "
Oct 30 19:32:55.287544 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:55.287466    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-openshift-ca\") pod \"8e91d125-96d0-4a51-acb8-bd6060ff2766\" (UID: \"8e91d125-96d0-4a51-acb8-bd6060ff2766\") "
Oct 30 19:32:55.287908 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:55.287880    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-openshift-ca" pod "8e91d125-96d0-4a51-acb8-bd6060ff2766" (UID: "8e91d125-96d0-4a51-acb8-bd6060ff2766"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:55.289389 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:55.289365    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-xks-ca" pod "8e91d125-96d0-4a51-acb8-bd6060ff2766" (UID: "8e91d125-96d0-4a51-acb8-bd6060ff2766"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:55.289481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:55.289452    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e91d125-96d0-4a51-acb8-bd6060ff2766-kube-api-access-ffzbd" pod "8e91d125-96d0-4a51-acb8-bd6060ff2766" (UID: "8e91d125-96d0-4a51-acb8-bd6060ff2766"). InnerVolumeSpecName "kube-api-access-ffzbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:55.388218 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:55.388137    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:32:55.388218 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:55.388165    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffzbd\" (UniqueName: \"kubernetes.io/projected/8e91d125-96d0-4a51-acb8-bd6060ff2766-kube-api-access-ffzbd\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:32:55.388218 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:55.388176    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e91d125-96d0-4a51-acb8-bd6060ff2766-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:32:56.094927 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:56.094888    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7" event={"ID":"8e91d125-96d0-4a51-acb8-bd6060ff2766","Type":"ContainerDied","Data":"c17f6d07d95d2eeea5e91986f36d283e8e7ce8e2857ca404c48a92cafc2b3b81"}
Oct 30 19:32:56.095122 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:56.094950    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c17f6d07d95d2eeea5e91986f36d283e8e7ce8e2857ca404c48a92cafc2b3b81"
Oct 30 19:32:56.095122 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:32:56.094901    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"
Oct 30 19:33:11.058880 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:11.058765    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"]
Oct 30 19:33:11.115325 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:11.115281    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s56dh"]
Oct 30 19:33:11.295261 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:11.294653    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324636a4-6291-44ac-bab9-90c212614090" path="/var/lib/kubelet/pods/324636a4-6291-44ac-bab9-90c212614090/volumes"
Oct 30 19:33:11.597522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:11.595739    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-864kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:33:11.597522 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:11.595860    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-864kk" podUID="96d47ea1-b235-45d8-8573-0dae95cd754d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:11.601551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:11.600609    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8kbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:33:11.601551 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:11.600697    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" podUID="38d93a40-3273-466a-9475-fa0960ed3e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:13.212756 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.212713    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"]
Oct 30 19:33:13.213252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.213069    4267 cpu_manager.go:414] "containerMap: removing container" podUID="8e91d125-96d0-4a51-acb8-bd6060ff2766" containerName="revoke-keys"
Oct 30 19:33:13.213252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.213080    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e91d125-96d0-4a51-acb8-bd6060ff2766" containerName="revoke-keys"
Oct 30 19:33:13.213252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.213170    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e91d125-96d0-4a51-acb8-bd6060ff2766" containerName="revoke-keys"
Oct 30 19:33:13.216541 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.216508    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.270984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.270940    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:33:13.320577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.320540    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:33:13.320789 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.320649    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:33:13.320789 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.320695    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:33:13.322989 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.322968    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"]
Oct 30 19:33:13.340831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.340797    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6snnd\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.340994 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.340931    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6snnd\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.341080 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.341058    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p6sx\" (UniqueName: \"kubernetes.io/projected/102f6129-2699-4328-8e3d-f1f11ebc5f99-kube-api-access-2p6sx\") pod \"maas-api-revoke-sub-d2bc93bc-6snnd\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.442416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.442379    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p6sx\" (UniqueName: \"kubernetes.io/projected/102f6129-2699-4328-8e3d-f1f11ebc5f99-kube-api-access-2p6sx\") pod \"maas-api-revoke-sub-d2bc93bc-6snnd\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.442673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.442440    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6snnd\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.442673 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.442480    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6snnd\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.443109 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.443083    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6snnd\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.444930 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.444900    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6snnd\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.499481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.499386    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p6sx\" (UniqueName: \"kubernetes.io/projected/102f6129-2699-4328-8e3d-f1f11ebc5f99-kube-api-access-2p6sx\") pod \"maas-api-revoke-sub-d2bc93bc-6snnd\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.530439 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.530402    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:13.901042 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:13.901009    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"]
Oct 30 19:33:13.902505 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:33:13.902457    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102f6129_2699_4328_8e3d_f1f11ebc5f99.slice/crio-555783e9fcda721fbd54872bca9d7bd7d6ea19a464477b48cfba0478cfa3d36e WatchSource:0}: Error finding container 555783e9fcda721fbd54872bca9d7bd7d6ea19a464477b48cfba0478cfa3d36e: Status 404 returned error can't find the container with id 555783e9fcda721fbd54872bca9d7bd7d6ea19a464477b48cfba0478cfa3d36e
Oct 30 19:33:14.228387 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:14.228361    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd" event={"ID":"102f6129-2699-4328-8e3d-f1f11ebc5f99","Type":"ContainerStarted","Data":"555783e9fcda721fbd54872bca9d7bd7d6ea19a464477b48cfba0478cfa3d36e"}
Oct 30 19:33:15.234377 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:15.234343    4267 generic.go:356] "Generic (PLEG): container finished" podID="102f6129-2699-4328-8e3d-f1f11ebc5f99" containerID="779c49d61557d368004a2003aeab1614fbb9219104da11628dd56f151b24ada2" exitCode=0
Oct 30 19:33:15.234798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:15.234379    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd" event={"ID":"102f6129-2699-4328-8e3d-f1f11ebc5f99","Type":"ContainerDied","Data":"779c49d61557d368004a2003aeab1614fbb9219104da11628dd56f151b24ada2"}
Oct 30 19:33:16.369067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:16.369039    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:16.470622 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:16.470594    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102f6129-2699-4328-8e3d-f1f11ebc5f99-kube-api-access-2p6sx\" (UniqueName: \"kubernetes.io/projected/102f6129-2699-4328-8e3d-f1f11ebc5f99-kube-api-access-2p6sx\") pod \"102f6129-2699-4328-8e3d-f1f11ebc5f99\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") "
Oct 30 19:33:16.470838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:16.470632    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-xks-ca\") pod \"102f6129-2699-4328-8e3d-f1f11ebc5f99\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") "
Oct 30 19:33:16.470838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:16.470696    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-openshift-ca\") pod \"102f6129-2699-4328-8e3d-f1f11ebc5f99\" (UID: \"102f6129-2699-4328-8e3d-f1f11ebc5f99\") "
Oct 30 19:33:16.471083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:16.471060    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-openshift-ca" pod "102f6129-2699-4328-8e3d-f1f11ebc5f99" (UID: "102f6129-2699-4328-8e3d-f1f11ebc5f99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:16.472631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:16.472599    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102f6129-2699-4328-8e3d-f1f11ebc5f99-kube-api-access-2p6sx" pod "102f6129-2699-4328-8e3d-f1f11ebc5f99" (UID: "102f6129-2699-4328-8e3d-f1f11ebc5f99"). InnerVolumeSpecName "kube-api-access-2p6sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:16.472740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:16.472717    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-xks-ca" pod "102f6129-2699-4328-8e3d-f1f11ebc5f99" (UID: "102f6129-2699-4328-8e3d-f1f11ebc5f99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:16.572220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:16.572156    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p6sx\" (UniqueName: \"kubernetes.io/projected/102f6129-2699-4328-8e3d-f1f11ebc5f99-kube-api-access-2p6sx\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:33:16.572220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:16.572185    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:33:16.572220 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:16.572197    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/102f6129-2699-4328-8e3d-f1f11ebc5f99-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:33:17.246240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:17.246210    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"
Oct 30 19:33:17.246240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:17.246219    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd" event={"ID":"102f6129-2699-4328-8e3d-f1f11ebc5f99","Type":"ContainerDied","Data":"555783e9fcda721fbd54872bca9d7bd7d6ea19a464477b48cfba0478cfa3d36e"}
Oct 30 19:33:17.246429 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:17.246263    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="555783e9fcda721fbd54872bca9d7bd7d6ea19a464477b48cfba0478cfa3d36e"
Oct 30 19:33:20.157999 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:33:20.157960    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:33:32.891864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:32.891833    4267 scope.go:122] "RemoveContainer" containerID="a451ee609e36b9e6e7ee33c41d159cb15d968d0d6d84f06f649db3f2b5f82d92"
Oct 30 19:33:32.900675 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:32.900645    4267 scope.go:122] "RemoveContainer" containerID="7aabc826dea85ef1007dca6aa0320056f43fb40e3bb3336018315eef94153433"
Oct 30 19:33:34.289672 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.289638    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"]
Oct 30 19:33:34.290077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.289970    4267 cpu_manager.go:414] "containerMap: removing container" podUID="102f6129-2699-4328-8e3d-f1f11ebc5f99" containerName="revoke-keys"
Oct 30 19:33:34.290077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.289980    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102f6129-2699-4328-8e3d-f1f11ebc5f99" containerName="revoke-keys"
Oct 30 19:33:34.290077 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.290054    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="102f6129-2699-4328-8e3d-f1f11ebc5f99" containerName="revoke-keys"
Oct 30 19:33:34.294471 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.294452    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:33:34.346370 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.346335    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-682a87\""
Oct 30 19:33:34.400426 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.400387    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"]
Oct 30 19:33:34.400635 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.400613    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e59c28d7-dockercfg-sx22v\""
Oct 30 19:33:34.444052 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.444020    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5bcb27-72ff-4836-b134-71376773c628-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz\" (UID: \"fc5bcb27-72ff-4836-b134-71376773c628\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:33:34.444227 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.444087    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zksnc\" (UniqueName: \"kubernetes.io/projected/fc5bcb27-72ff-4836-b134-71376773c628-kube-api-access-zksnc\") pod \"payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz\" (UID: \"fc5bcb27-72ff-4836-b134-71376773c628\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:33:34.448393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.448365    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"]
Oct 30 19:33:34.451775 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.451759    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:33:34.501252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.501220    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"]
Oct 30 19:33:34.545015 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.544935    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5bcb27-72ff-4836-b134-71376773c628-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz\" (UID: \"fc5bcb27-72ff-4836-b134-71376773c628\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:33:34.545015 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.544982    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc880e71-1157-4417-9a96-47749908a0ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv\" (UID: \"fc880e71-1157-4417-9a96-47749908a0ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:33:34.545205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.545012    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv9hs\" (UniqueName: \"kubernetes.io/projected/fc880e71-1157-4417-9a96-47749908a0ef-kube-api-access-lv9hs\") pod \"payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv\" (UID: \"fc880e71-1157-4417-9a96-47749908a0ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:33:34.545205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.545072    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zksnc\" (UniqueName: \"kubernetes.io/projected/fc5bcb27-72ff-4836-b134-71376773c628-kube-api-access-zksnc\") pod \"payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz\" (UID: \"fc5bcb27-72ff-4836-b134-71376773c628\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:33:34.545617 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.545598    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5bcb27-72ff-4836-b134-71376773c628-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz\" (UID: \"fc5bcb27-72ff-4836-b134-71376773c628\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:33:34.601733 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.601705    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zksnc\" (UniqueName: \"kubernetes.io/projected/fc5bcb27-72ff-4836-b134-71376773c628-kube-api-access-zksnc\") pod \"payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz\" (UID: \"fc5bcb27-72ff-4836-b134-71376773c628\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:33:34.606568 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.606549    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:33:34.645961 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.645925    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc880e71-1157-4417-9a96-47749908a0ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv\" (UID: \"fc880e71-1157-4417-9a96-47749908a0ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:33:34.645961 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.645966    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv9hs\" (UniqueName: \"kubernetes.io/projected/fc880e71-1157-4417-9a96-47749908a0ef-kube-api-access-lv9hs\") pod \"payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv\" (UID: \"fc880e71-1157-4417-9a96-47749908a0ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:33:34.646670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.646647    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc880e71-1157-4417-9a96-47749908a0ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv\" (UID: \"fc880e71-1157-4417-9a96-47749908a0ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:33:34.703120 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.703091    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv9hs\" (UniqueName: \"kubernetes.io/projected/fc880e71-1157-4417-9a96-47749908a0ef-kube-api-access-lv9hs\") pod \"payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv\" (UID: \"fc880e71-1157-4417-9a96-47749908a0ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:33:34.764311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.764284    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:33:34.980311 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:34.980283    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"]
Oct 30 19:33:34.981792 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:33:34.981766    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5bcb27_72ff_4836_b134_71376773c628.slice/crio-7b0a1b8b0ac209f95c9d5de0ee341514d825ffffbc9637affc067b7e47a07d2a WatchSource:0}: Error finding container 7b0a1b8b0ac209f95c9d5de0ee341514d825ffffbc9637affc067b7e47a07d2a: Status 404 returned error can't find the container with id 7b0a1b8b0ac209f95c9d5de0ee341514d825ffffbc9637affc067b7e47a07d2a
Oct 30 19:33:35.136617 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:35.136593    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"]
Oct 30 19:33:35.138094 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:33:35.138068    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc880e71_1157_4417_9a96_47749908a0ef.slice/crio-0b284370df042e38fc4fbfdef8ecc2ada9a0f6613a00ca36b1f0c1ac69481021 WatchSource:0}: Error finding container 0b284370df042e38fc4fbfdef8ecc2ada9a0f6613a00ca36b1f0c1ac69481021: Status 404 returned error can't find the container with id 0b284370df042e38fc4fbfdef8ecc2ada9a0f6613a00ca36b1f0c1ac69481021
Oct 30 19:33:35.345104 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:35.345071    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv" event={"ID":"fc880e71-1157-4417-9a96-47749908a0ef","Type":"ContainerStarted","Data":"b4105c546a73e628112745893f40ce0549ac4816d1f15dc6e1b638810f53a1d3"}
Oct 30 19:33:35.345558 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:35.345123    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv" event={"ID":"fc880e71-1157-4417-9a96-47749908a0ef","Type":"ContainerStarted","Data":"0b284370df042e38fc4fbfdef8ecc2ada9a0f6613a00ca36b1f0c1ac69481021"}
Oct 30 19:33:35.345558 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:35.345237    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:33:35.346809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:35.346785    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz" event={"ID":"fc5bcb27-72ff-4836-b134-71376773c628","Type":"ContainerStarted","Data":"e2ab91810f3406fe272ab1e03a248886dbc5d06289ce3122b401a5d0b26884d1"}
Oct 30 19:33:35.346920 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:35.346821    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz" event={"ID":"fc5bcb27-72ff-4836-b134-71376773c628","Type":"ContainerStarted","Data":"7b0a1b8b0ac209f95c9d5de0ee341514d825ffffbc9637affc067b7e47a07d2a"}
Oct 30 19:33:35.346970 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:35.346920    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:33:35.463679 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:35.463636    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv" podStartSLOduration=1.463623734 podStartE2EDuration="1.463623734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:35.461720945 +0000 UTC m=+3306.349638035" watchObservedRunningTime="2026-10-30 19:33:35.463623734 +0000 UTC m=+3306.351540822"
Oct 30 19:33:35.590307 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:35.586371    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz" podStartSLOduration=1.586350671 podStartE2EDuration="1.586350671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:35.5826386 +0000 UTC m=+3306.470555685" watchObservedRunningTime="2026-10-30 19:33:35.586350671 +0000 UTC m=+3306.474267762"
Oct 30 19:33:46.355069 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:46.355041    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:33:46.355602 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:46.355199    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:33:50.156756 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:33:50.156718    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:33:59.130561 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.130520    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"]
Oct 30 19:33:59.135240 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.135220    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.183801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.183759    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f072baa-de78-4443-b9b1-251b3d13886a-tls-certs\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.183984 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.183829    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6w69\" (UniqueName: \"kubernetes.io/projected/9f072baa-de78-4443-b9b1-251b3d13886a-kube-api-access-m6w69\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.184038 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.183980    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-home\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.184130 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.184102    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-model-cache\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.184198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.184183    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-tmp-dir\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.184265 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.184251    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-dshm\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.238423 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.238391    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-682a87\"/\"kube-root-ca.crt\""
Oct 30 19:33:59.238670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.238556    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-682a87\"/\"openshift-service-ca.crt\""
Oct 30 19:33:59.238798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.238777    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-682a87\"/\"simulator-single-main-682a87-kserve-self-signed-certs\""
Oct 30 19:33:59.238910 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.238891    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-682a87\"/\"default-dockercfg-mf62j\""
Oct 30 19:33:59.241116 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.241098    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"]
Oct 30 19:33:59.285079 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.285041    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-model-cache\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.285301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.285112    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-tmp-dir\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.285301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.285162    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-dshm\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.285301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.285195    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f072baa-de78-4443-b9b1-251b3d13886a-tls-certs\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.285301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.285220    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6w69\" (UniqueName: \"kubernetes.io/projected/9f072baa-de78-4443-b9b1-251b3d13886a-kube-api-access-m6w69\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.285301 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.285292    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-home\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.285643 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.285614    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-model-cache\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.285720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.285695    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-tmp-dir\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.287205 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.287175    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-home\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.290758 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.290730    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f072baa-de78-4443-b9b1-251b3d13886a-tls-certs\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.295150 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.295123    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-dshm\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.345951 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.345908    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6w69\" (UniqueName: \"kubernetes.io/projected/9f072baa-de78-4443-b9b1-251b3d13886a-kube-api-access-m6w69\") pod \"simulator-single-main-682a87-kserve-fcd88c54d-9pk9q\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") " pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.447354 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.447319    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:33:59.820317 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:33:59.820290    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"]
Oct 30 19:33:59.821682 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:33:59.821653    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f072baa_de78_4443_b9b1_251b3d13886a.slice/crio-44ae1b20a23bb704c78bfebef10a40098eaf164ab439f545c1e7e3fcacdbc5f8 WatchSource:0}: Error finding container 44ae1b20a23bb704c78bfebef10a40098eaf164ab439f545c1e7e3fcacdbc5f8: Status 404 returned error can't find the container with id 44ae1b20a23bb704c78bfebef10a40098eaf164ab439f545c1e7e3fcacdbc5f8
Oct 30 19:34:00.483910 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:00.483859    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q" event={"ID":"9f072baa-de78-4443-b9b1-251b3d13886a","Type":"ContainerStarted","Data":"7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56"}
Oct 30 19:34:00.484366 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:00.483919    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q" event={"ID":"9f072baa-de78-4443-b9b1-251b3d13886a","Type":"ContainerStarted","Data":"44ae1b20a23bb704c78bfebef10a40098eaf164ab439f545c1e7e3fcacdbc5f8"}
Oct 30 19:34:00.601253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:00.601194    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q" podStartSLOduration=1.601178311 podStartE2EDuration="1.601178311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:00.598990697 +0000 UTC m=+3331.486907791" watchObservedRunningTime="2026-10-30 19:34:00.601178311 +0000 UTC m=+3331.489095399"
Oct 30 19:34:01.447720 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:01.447678    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:34:09.447757 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:09.447705    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:34:09.460010 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:09.459984    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:34:09.546121 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:09.546091    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:34:20.157720 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:34:20.157681    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:34:29.485200 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.485162    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"]
Oct 30 19:34:29.489166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.489147    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:29.542801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.542756    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:34:29.542801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.542788    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:29.578053 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.578019    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f0d1ad8-xrcbm\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:29.578230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.578096    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f0d1ad8-xrcbm\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:29.578230 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.578165    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6qlz\" (UniqueName: \"kubernetes.io/projected/5bc70997-f350-45bc-bdb7-8f32e8c75b11-kube-api-access-c6qlz\") pod \"maas-api-revoke-sub-9f0d1ad8-xrcbm\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:29.592186 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.592153    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:34:29.592346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.592269    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:34:29.595064 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.595040    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"]
Oct 30 19:34:29.679057 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.679013    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f0d1ad8-xrcbm\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:29.679280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.679126    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f0d1ad8-xrcbm\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:29.679280 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.679181    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6qlz\" (UniqueName: \"kubernetes.io/projected/5bc70997-f350-45bc-bdb7-8f32e8c75b11-kube-api-access-c6qlz\") pod \"maas-api-revoke-sub-9f0d1ad8-xrcbm\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:29.679803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.679777    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f0d1ad8-xrcbm\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:29.681462 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.681446    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f0d1ad8-xrcbm\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:29.736823 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.736754    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6qlz\" (UniqueName: \"kubernetes.io/projected/5bc70997-f350-45bc-bdb7-8f32e8c75b11-kube-api-access-c6qlz\") pod \"maas-api-revoke-sub-9f0d1ad8-xrcbm\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:29.803035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:29.802992    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:30.172906 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:30.172874    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"]
Oct 30 19:34:30.173998 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:34:30.173970    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc70997_f350_45bc_bdb7_8f32e8c75b11.slice/crio-ec69a4760d53f063d3ba313685f9a3cf574d674a24c9d6af22b93a1235da9213 WatchSource:0}: Error finding container ec69a4760d53f063d3ba313685f9a3cf574d674a24c9d6af22b93a1235da9213: Status 404 returned error can't find the container with id ec69a4760d53f063d3ba313685f9a3cf574d674a24c9d6af22b93a1235da9213
Oct 30 19:34:30.656152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:30.656109    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm" event={"ID":"5bc70997-f350-45bc-bdb7-8f32e8c75b11","Type":"ContainerStarted","Data":"6012efe150c06006fd0cd3d2cf5692a31e078fa78140c4e77d8744ac86e026be"}
Oct 30 19:34:30.656670 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:30.656170    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm" event={"ID":"5bc70997-f350-45bc-bdb7-8f32e8c75b11","Type":"ContainerStarted","Data":"ec69a4760d53f063d3ba313685f9a3cf574d674a24c9d6af22b93a1235da9213"}
Oct 30 19:34:30.772621 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:30.772565    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm" podStartSLOduration=1.772545911 podStartE2EDuration="1.772545911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:30.77195512 +0000 UTC m=+3361.659872210" watchObservedRunningTime="2026-10-30 19:34:30.772545911 +0000 UTC m=+3361.660463000"
Oct 30 19:34:31.663088 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:31.663048    4267 generic.go:356] "Generic (PLEG): container finished" podID="5bc70997-f350-45bc-bdb7-8f32e8c75b11" containerID="6012efe150c06006fd0cd3d2cf5692a31e078fa78140c4e77d8744ac86e026be" exitCode=0
Oct 30 19:34:31.663548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:31.663134    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm" event={"ID":"5bc70997-f350-45bc-bdb7-8f32e8c75b11","Type":"ContainerDied","Data":"6012efe150c06006fd0cd3d2cf5692a31e078fa78140c4e77d8744ac86e026be"}
Oct 30 19:34:32.806502 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.806459    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:32.911577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.911536    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-openshift-ca\") pod \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") "
Oct 30 19:34:32.911793 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.911620    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-xks-ca\") pod \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") "
Oct 30 19:34:32.911793 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.911726    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc70997-f350-45bc-bdb7-8f32e8c75b11-kube-api-access-c6qlz\" (UniqueName: \"kubernetes.io/projected/5bc70997-f350-45bc-bdb7-8f32e8c75b11-kube-api-access-c6qlz\") pod \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\" (UID: \"5bc70997-f350-45bc-bdb7-8f32e8c75b11\") "
Oct 30 19:34:32.911997 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.911974    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-openshift-ca" pod "5bc70997-f350-45bc-bdb7-8f32e8c75b11" (UID: "5bc70997-f350-45bc-bdb7-8f32e8c75b11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:32.912165 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.912149    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.913791 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.913764    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-xks-ca" pod "5bc70997-f350-45bc-bdb7-8f32e8c75b11" (UID: "5bc70997-f350-45bc-bdb7-8f32e8c75b11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:32.913903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.913768    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc70997-f350-45bc-bdb7-8f32e8c75b11-kube-api-access-c6qlz" pod "5bc70997-f350-45bc-bdb7-8f32e8c75b11" (UID: "5bc70997-f350-45bc-bdb7-8f32e8c75b11"). InnerVolumeSpecName "kube-api-access-c6qlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:32.928009 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.927947    4267 scope.go:122] "RemoveContainer" containerID="2774c24698272c95c17de3f7464f9633e3bc7da32a042f82226278d849ea6614"
Oct 30 19:34:32.938605 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.938583    4267 scope.go:122] "RemoveContainer" containerID="18f8483ac36d880f8002ba922af3a6ee3c6b559edee27dd16ff0b84f57771bf8"
Oct 30 19:34:32.947563 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:32.947536    4267 scope.go:122] "RemoveContainer" containerID="925453b4c674d04bff5d80465a14a61da520c315f39ba10e77fbd1e3875f806b"
Oct 30 19:34:33.013313 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:33.013282    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc70997-f350-45bc-bdb7-8f32e8c75b11-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:34:33.013439 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:33.013320    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6qlz\" (UniqueName: \"kubernetes.io/projected/5bc70997-f350-45bc-bdb7-8f32e8c75b11-kube-api-access-c6qlz\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:34:33.676170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:33.676095    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"
Oct 30 19:34:33.676170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:33.676118    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm" event={"ID":"5bc70997-f350-45bc-bdb7-8f32e8c75b11","Type":"ContainerDied","Data":"ec69a4760d53f063d3ba313685f9a3cf574d674a24c9d6af22b93a1235da9213"}
Oct 30 19:34:33.676170 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:33.676161    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec69a4760d53f063d3ba313685f9a3cf574d674a24c9d6af22b93a1235da9213"
Oct 30 19:34:40.836348 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.836306    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"]
Oct 30 19:34:40.836776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.836706    4267 cpu_manager.go:414] "containerMap: removing container" podUID="5bc70997-f350-45bc-bdb7-8f32e8c75b11" containerName="revoke-keys"
Oct 30 19:34:40.836776 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.836721    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc70997-f350-45bc-bdb7-8f32e8c75b11" containerName="revoke-keys"
Oct 30 19:34:40.836857 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.836799    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc70997-f350-45bc-bdb7-8f32e8c75b11" containerName="revoke-keys"
Oct 30 19:34:40.841247 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.841225    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:40.883914 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.883877    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:40.884129 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.883940    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:40.884129 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.884057    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfbdq\" (UniqueName: \"kubernetes.io/projected/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-kube-api-access-tfbdq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:40.944826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.944783    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:40.944826 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.944810    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:34:40.945105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.944861    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:34:40.945105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.944878    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:34:40.947631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.947607    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"]
Oct 30 19:34:40.985435 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.985359    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:40.985435 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.985403    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfbdq\" (UniqueName: \"kubernetes.io/projected/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-kube-api-access-tfbdq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:40.985648 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.985476    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:40.986105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.986083    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:40.988321 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:40.988301    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:41.043651 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:41.043611    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfbdq\" (UniqueName: \"kubernetes.io/projected/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-kube-api-access-tfbdq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:41.154885 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:41.154787    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:41.543780 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:34:41.543112    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fa0c84_6ea2_4c3d_b54f_868b2bf10fdf.slice/crio-a6d80dfb63d395164959054d59e897b01fac7174275e74f7df4120d5820fbc6b WatchSource:0}: Error finding container a6d80dfb63d395164959054d59e897b01fac7174275e74f7df4120d5820fbc6b: Status 404 returned error can't find the container with id a6d80dfb63d395164959054d59e897b01fac7174275e74f7df4120d5820fbc6b
Oct 30 19:34:41.543780 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:41.543586    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"]
Oct 30 19:34:41.722925 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:41.722891    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb" event={"ID":"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf","Type":"ContainerStarted","Data":"a6d80dfb63d395164959054d59e897b01fac7174275e74f7df4120d5820fbc6b"}
Oct 30 19:34:42.730133 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:42.730097    4267 generic.go:356] "Generic (PLEG): container finished" podID="b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf" containerID="0613b274496466f069d76848727a6a7a689f8889dfeac80907580fa107421d8e" exitCode=0
Oct 30 19:34:42.730133 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:42.730137    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb" event={"ID":"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf","Type":"ContainerDied","Data":"0613b274496466f069d76848727a6a7a689f8889dfeac80907580fa107421d8e"}
Oct 30 19:34:43.880761 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:43.880727    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:44.014449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.014364    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-openshift-ca\") pod \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") "
Oct 30 19:34:44.014449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.014411    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-xks-ca\") pod \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") "
Oct 30 19:34:44.014674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.014469    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-kube-api-access-tfbdq\" (UniqueName: \"kubernetes.io/projected/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-kube-api-access-tfbdq\") pod \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\" (UID: \"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf\") "
Oct 30 19:34:44.014827 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.014802    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-openshift-ca" pod "b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf" (UID: "b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:44.016631 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.016599    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-xks-ca" pod "b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf" (UID: "b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:44.016780 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.016759    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-kube-api-access-tfbdq" pod "b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf" (UID: "b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf"). InnerVolumeSpecName "kube-api-access-tfbdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:44.115937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.115901    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:34:44.115937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.115932    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:34:44.115937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.115941    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfbdq\" (UniqueName: \"kubernetes.io/projected/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf-kube-api-access-tfbdq\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:34:44.743312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.743280    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"
Oct 30 19:34:44.743312 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.743306    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb" event={"ID":"b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf","Type":"ContainerDied","Data":"a6d80dfb63d395164959054d59e897b01fac7174275e74f7df4120d5820fbc6b"}
Oct 30 19:34:44.743628 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:44.743358    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6d80dfb63d395164959054d59e897b01fac7174275e74f7df4120d5820fbc6b"
Oct 30 19:34:45.522166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:45.522124    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"]
Oct 30 19:34:50.158157 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:34:50.158116    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:34:52.273152 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:52.273116    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"]
Oct 30 19:34:52.379024 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:52.378986    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"]
Oct 30 19:34:57.274369 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:57.274284    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv" podUID="fc880e71-1157-4417-9a96-47749908a0ef" containerName="payload-processing" containerID="cri-o://b4105c546a73e628112745893f40ce0549ac4816d1f15dc6e1b638810f53a1d3" gracePeriod=25
Oct 30 19:34:57.291332 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:57.291293    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"]
Oct 30 19:34:57.340706 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:57.340672    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-682a87-2pkfb"]
Oct 30 19:34:57.380118 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:57.380041    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz" podUID="fc5bcb27-72ff-4836-b134-71376773c628" containerName="payload-pre-processing" containerID="cri-o://e2ab91810f3406fe272ab1e03a248886dbc5d06289ce3122b401a5d0b26884d1" gracePeriod=25
Oct 30 19:34:59.219917 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:34:59.219879    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf" path="/var/lib/kubelet/pods/b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf/volumes"
Oct 30 19:35:00.544292 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.544227    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q" podUID="9f072baa-de78-4443-b9b1-251b3d13886a" containerName="main" containerID="cri-o://7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56" gracePeriod=45
Oct 30 19:35:00.800809 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.800740    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:35:00.829452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.829413    4267 generic.go:356] "Generic (PLEG): container finished" podID="9f072baa-de78-4443-b9b1-251b3d13886a" containerID="7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56" exitCode=0
Oct 30 19:35:00.829659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.829466    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"
Oct 30 19:35:00.829659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.829506    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q" event={"ID":"9f072baa-de78-4443-b9b1-251b3d13886a","Type":"ContainerDied","Data":"7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56"}
Oct 30 19:35:00.829659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.829550    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q" event={"ID":"9f072baa-de78-4443-b9b1-251b3d13886a","Type":"ContainerDied","Data":"44ae1b20a23bb704c78bfebef10a40098eaf164ab439f545c1e7e3fcacdbc5f8"}
Oct 30 19:35:00.829659 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.829573    4267 scope.go:122] "RemoveContainer" containerID="7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56"
Oct 30 19:35:00.838294 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.838276    4267 scope.go:122] "RemoveContainer" containerID="7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56"
Oct 30 19:35:00.838612 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:35:00.838594    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56\": container with ID starting with 7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56 not found: ID does not exist" containerID="7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56"
Oct 30 19:35:00.838669 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.838624    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56"} err="failed to get container status \"7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56\": rpc error: code = NotFound desc = could not find container \"7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56\": container with ID starting with 7955aa19c44668e037c589d4719d1565ec28cabe37af52e306f7b42b62093d56 not found: ID does not exist"
Oct 30 19:35:00.888336 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888300    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f072baa-de78-4443-b9b1-251b3d13886a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f072baa-de78-4443-b9b1-251b3d13886a-tls-certs\") pod \"9f072baa-de78-4443-b9b1-251b3d13886a\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") "
Oct 30 19:35:00.888553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888346    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-home\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-home\") pod \"9f072baa-de78-4443-b9b1-251b3d13886a\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") "
Oct 30 19:35:00.888553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888378    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-dshm\") pod \"9f072baa-de78-4443-b9b1-251b3d13886a\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") "
Oct 30 19:35:00.888553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888397    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-tmp-dir\") pod \"9f072baa-de78-4443-b9b1-251b3d13886a\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") "
Oct 30 19:35:00.888553 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888421    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f072baa-de78-4443-b9b1-251b3d13886a-kube-api-access-m6w69\" (UniqueName: \"kubernetes.io/projected/9f072baa-de78-4443-b9b1-251b3d13886a-kube-api-access-m6w69\") pod \"9f072baa-de78-4443-b9b1-251b3d13886a\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") "
Oct 30 19:35:00.888798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888563    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-model-cache\") pod \"9f072baa-de78-4443-b9b1-251b3d13886a\" (UID: \"9f072baa-de78-4443-b9b1-251b3d13886a\") "
Oct 30 19:35:00.888798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888702    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-tmp-dir" pod "9f072baa-de78-4443-b9b1-251b3d13886a" (UID: "9f072baa-de78-4443-b9b1-251b3d13886a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:00.888798 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888723    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-home" pod "9f072baa-de78-4443-b9b1-251b3d13886a" (UID: "9f072baa-de78-4443-b9b1-251b3d13886a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:00.888967 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888883    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-model-cache" pod "9f072baa-de78-4443-b9b1-251b3d13886a" (UID: "9f072baa-de78-4443-b9b1-251b3d13886a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:00.888967 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888954    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:00.889084 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888969    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:00.889084 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.888981    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:00.890853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.890823    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f072baa-de78-4443-b9b1-251b3d13886a-tls-certs" pod "9f072baa-de78-4443-b9b1-251b3d13886a" (UID: "9f072baa-de78-4443-b9b1-251b3d13886a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:00.890853 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.890838    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-dshm" pod "9f072baa-de78-4443-b9b1-251b3d13886a" (UID: "9f072baa-de78-4443-b9b1-251b3d13886a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:00.891003 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.890866    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f072baa-de78-4443-b9b1-251b3d13886a-kube-api-access-m6w69" pod "9f072baa-de78-4443-b9b1-251b3d13886a" (UID: "9f072baa-de78-4443-b9b1-251b3d13886a"). InnerVolumeSpecName "kube-api-access-m6w69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:00.990315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.990278    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f072baa-de78-4443-b9b1-251b3d13886a-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:00.990315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.990312    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6w69\" (UniqueName: \"kubernetes.io/projected/9f072baa-de78-4443-b9b1-251b3d13886a-kube-api-access-m6w69\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:00.990315 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:00.990322    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f072baa-de78-4443-b9b1-251b3d13886a-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:01.250825 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:01.250785    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"]
Oct 30 19:35:01.254582 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:01.254552    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-682a87/simulator-single-main-682a87-kserve-fcd88c54d-9pk9q"]
Oct 30 19:35:03.220031 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:03.219995    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f072baa-de78-4443-b9b1-251b3d13886a" path="/var/lib/kubelet/pods/9f072baa-de78-4443-b9b1-251b3d13886a/volumes"
Oct 30 19:35:06.353841 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:06.353803    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv" podUID="fc880e71-1157-4417-9a96-47749908a0ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:06.354250 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:06.353845    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz" podUID="fc5bcb27-72ff-4836-b134-71376773c628" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:16.354531 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:16.354106    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz" podUID="fc5bcb27-72ff-4836-b134-71376773c628" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:16.355047 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:16.354966    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv" podUID="fc880e71-1157-4417-9a96-47749908a0ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:18.021403 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.021124    4267 generic.go:356] "Generic (PLEG): container finished" podID="fc5bcb27-72ff-4836-b134-71376773c628" containerID="e2ab91810f3406fe272ab1e03a248886dbc5d06289ce3122b401a5d0b26884d1" exitCode=0
Oct 30 19:35:18.021403 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.021319    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz" event={"ID":"fc5bcb27-72ff-4836-b134-71376773c628","Type":"ContainerDied","Data":"e2ab91810f3406fe272ab1e03a248886dbc5d06289ce3122b401a5d0b26884d1"}
Oct 30 19:35:18.067513 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.064317    4267 generic.go:356] "Generic (PLEG): container finished" podID="fc880e71-1157-4417-9a96-47749908a0ef" containerID="b4105c546a73e628112745893f40ce0549ac4816d1f15dc6e1b638810f53a1d3" exitCode=0
Oct 30 19:35:18.069472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.069438    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv" event={"ID":"fc880e71-1157-4417-9a96-47749908a0ef","Type":"ContainerDied","Data":"b4105c546a73e628112745893f40ce0549ac4816d1f15dc6e1b638810f53a1d3"}
Oct 30 19:35:18.305019 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.304858    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:35:18.429358 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.429332    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5bcb27-72ff-4836-b134-71376773c628-kube-api-access-zksnc\" (UniqueName: \"kubernetes.io/projected/fc5bcb27-72ff-4836-b134-71376773c628-kube-api-access-zksnc\") pod \"fc5bcb27-72ff-4836-b134-71376773c628\" (UID: \"fc5bcb27-72ff-4836-b134-71376773c628\") "
Oct 30 19:35:18.429549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.429531    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5bcb27-72ff-4836-b134-71376773c628-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5bcb27-72ff-4836-b134-71376773c628-plugins-config-volume\") pod \"fc5bcb27-72ff-4836-b134-71376773c628\" (UID: \"fc5bcb27-72ff-4836-b134-71376773c628\") "
Oct 30 19:35:18.445051 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.444052    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5bcb27-72ff-4836-b134-71376773c628-kube-api-access-zksnc" pod "fc5bcb27-72ff-4836-b134-71376773c628" (UID: "fc5bcb27-72ff-4836-b134-71376773c628"). InnerVolumeSpecName "kube-api-access-zksnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:18.445552 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.445521    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5bcb27-72ff-4836-b134-71376773c628-plugins-config-volume" pod "fc5bcb27-72ff-4836-b134-71376773c628" (UID: "fc5bcb27-72ff-4836-b134-71376773c628"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:18.550534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.550475    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zksnc\" (UniqueName: \"kubernetes.io/projected/fc5bcb27-72ff-4836-b134-71376773c628-kube-api-access-zksnc\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:18.550801 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.550778    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5bcb27-72ff-4836-b134-71376773c628-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:18.656577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.635289    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:35:18.763517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.762831    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc880e71-1157-4417-9a96-47749908a0ef-kube-api-access-lv9hs\" (UniqueName: \"kubernetes.io/projected/fc880e71-1157-4417-9a96-47749908a0ef-kube-api-access-lv9hs\") pod \"fc880e71-1157-4417-9a96-47749908a0ef\" (UID: \"fc880e71-1157-4417-9a96-47749908a0ef\") "
Oct 30 19:35:18.763517 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.762913    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc880e71-1157-4417-9a96-47749908a0ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc880e71-1157-4417-9a96-47749908a0ef-plugins-config-volume\") pod \"fc880e71-1157-4417-9a96-47749908a0ef\" (UID: \"fc880e71-1157-4417-9a96-47749908a0ef\") "
Oct 30 19:35:18.763803 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.763671    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc880e71-1157-4417-9a96-47749908a0ef-plugins-config-volume" pod "fc880e71-1157-4417-9a96-47749908a0ef" (UID: "fc880e71-1157-4417-9a96-47749908a0ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:18.787089 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.787049    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc880e71-1157-4417-9a96-47749908a0ef-kube-api-access-lv9hs" pod "fc880e71-1157-4417-9a96-47749908a0ef" (UID: "fc880e71-1157-4417-9a96-47749908a0ef"). InnerVolumeSpecName "kube-api-access-lv9hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:18.867125 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.867080    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv9hs\" (UniqueName: \"kubernetes.io/projected/fc880e71-1157-4417-9a96-47749908a0ef-kube-api-access-lv9hs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:18.867361 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:18.867340    4267 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc880e71-1157-4417-9a96-47749908a0ef-plugins-config-volume\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:35:19.105516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:19.100799    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv" event={"ID":"fc880e71-1157-4417-9a96-47749908a0ef","Type":"ContainerDied","Data":"0b284370df042e38fc4fbfdef8ecc2ada9a0f6613a00ca36b1f0c1ac69481021"}
Oct 30 19:35:19.105516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:19.100874    4267 scope.go:122] "RemoveContainer" containerID="b4105c546a73e628112745893f40ce0549ac4816d1f15dc6e1b638810f53a1d3"
Oct 30 19:35:19.105516 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:19.101025    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"
Oct 30 19:35:19.151879 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:19.151848    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz" event={"ID":"fc5bcb27-72ff-4836-b134-71376773c628","Type":"ContainerDied","Data":"7b0a1b8b0ac209f95c9d5de0ee341514d825ffffbc9637affc067b7e47a07d2a"}
Oct 30 19:35:19.156587 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:19.152035    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"
Oct 30 19:35:19.235534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:19.233650    4267 scope.go:122] "RemoveContainer" containerID="e2ab91810f3406fe272ab1e03a248886dbc5d06289ce3122b401a5d0b26884d1"
Oct 30 19:35:19.407649 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:19.407549    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"]
Oct 30 19:35:19.407969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:19.407944    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-682a87-597dbd757z47wv"]
Oct 30 19:35:19.507201 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:19.507168    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"]
Oct 30 19:35:19.541149 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:19.534148    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-682a87-dc67bd8jcz"]
Oct 30 19:35:20.157067 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:35:20.157028    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:35:21.218856 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:21.218823    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5bcb27-72ff-4836-b134-71376773c628" path="/var/lib/kubelet/pods/fc5bcb27-72ff-4836-b134-71376773c628/volumes"
Oct 30 19:35:21.219249 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:21.219216    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc880e71-1157-4417-9a96-47749908a0ef" path="/var/lib/kubelet/pods/fc880e71-1157-4417-9a96-47749908a0ef/volumes"
Oct 30 19:35:28.046323 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:28.046283    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"]
Oct 30 19:35:28.096143 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:28.096105    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c496f6-jgc42"]
Oct 30 19:35:29.219937 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:29.219907    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37282a31-72c0-451e-b2cb-bde8b50c6c95" path="/var/lib/kubelet/pods/37282a31-72c0-451e-b2cb-bde8b50c6c95/volumes"
Oct 30 19:35:32.980483 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:32.980449    4267 scope.go:122] "RemoveContainer" containerID="8bc8c35903b4781d7cf659ce5f096a0b27e69fffe5df7fd0a3e53ab0a6d02df3"
Oct 30 19:35:32.989156 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:32.989137    4267 scope.go:122] "RemoveContainer" containerID="5e97569acdf7a67809e2ba38c6bb336986e70d1ca13e4225a9c5c43bd0094e06"
Oct 30 19:35:50.158039 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:35:50.157996    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:35:54.472869 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.472826    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"]
Oct 30 19:35:54.473380 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473357    4267 cpu_manager.go:414] "containerMap: removing container" podUID="fc880e71-1157-4417-9a96-47749908a0ef" containerName="payload-processing"
Oct 30 19:35:54.473438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473386    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc880e71-1157-4417-9a96-47749908a0ef" containerName="payload-processing"
Oct 30 19:35:54.473438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473412    4267 cpu_manager.go:414] "containerMap: removing container" podUID="9f072baa-de78-4443-b9b1-251b3d13886a" containerName="main"
Oct 30 19:35:54.473438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473420    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f072baa-de78-4443-b9b1-251b3d13886a" containerName="main"
Oct 30 19:35:54.473438 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473434    4267 cpu_manager.go:414] "containerMap: removing container" podUID="b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf" containerName="revoke-keys"
Oct 30 19:35:54.473609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473442    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf" containerName="revoke-keys"
Oct 30 19:35:54.473609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473457    4267 cpu_manager.go:414] "containerMap: removing container" podUID="fc5bcb27-72ff-4836-b134-71376773c628" containerName="payload-pre-processing"
Oct 30 19:35:54.473609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473465    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5bcb27-72ff-4836-b134-71376773c628" containerName="payload-pre-processing"
Oct 30 19:35:54.473609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473588    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f072baa-de78-4443-b9b1-251b3d13886a" containerName="main"
Oct 30 19:35:54.473609 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473603    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5fa0c84-6ea2-4c3d-b54f-868b2bf10fdf" containerName="revoke-keys"
Oct 30 19:35:54.473772 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473611    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5bcb27-72ff-4836-b134-71376773c628" containerName="payload-pre-processing"
Oct 30 19:35:54.473772 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.473629    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc880e71-1157-4417-9a96-47749908a0ef" containerName="payload-processing"
Oct 30 19:35:54.478324 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.478305    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.579413 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.579379    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbg8q\" (UniqueName: \"kubernetes.io/projected/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-kube-api-access-dbg8q\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.579611 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.579426    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-home\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.579611 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.579551    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-model-cache\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.579696 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.579658    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-dshm\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.579742 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.579725    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tls-certs\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.579783 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.579764    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tmp-dir\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.581487 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.581466    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-00230e87-kserve-self-signed-certs\""
Oct 30 19:35:54.583788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.583762    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"]
Oct 30 19:35:54.680521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.680460    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-model-cache\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.680719 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.680549    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-dshm\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.680719 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.680598    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tls-certs\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.680719 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.680621    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tmp-dir\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.680719 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.680665    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbg8q\" (UniqueName: \"kubernetes.io/projected/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-kube-api-access-dbg8q\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.680719 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.680708    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-home\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.681009 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.680915    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-model-cache\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.681079 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.681008    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-home\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.681147 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.681095    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tmp-dir\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.682938 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.682917    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-dshm\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.683194 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.683176    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tls-certs\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.738434 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.738362    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbg8q\" (UniqueName: \"kubernetes.io/projected/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-kube-api-access-dbg8q\") pod \"e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") " pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:54.791740 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:54.791700    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:35:55.164677 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:55.164650    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"]
Oct 30 19:35:55.166411 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:35:55.166379    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e7763d2_08f8_4de1_ad2b_b7838b9f881b.slice/crio-46bbccbe12e69b95932728c18c33ead339b4eceb7454256b06eadef425197abf WatchSource:0}: Error finding container 46bbccbe12e69b95932728c18c33ead339b4eceb7454256b06eadef425197abf: Status 404 returned error can't find the container with id 46bbccbe12e69b95932728c18c33ead339b4eceb7454256b06eadef425197abf
Oct 30 19:35:55.351794 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:55.351759    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc" event={"ID":"8e7763d2-08f8-4de1-ad2b-b7838b9f881b","Type":"ContainerStarted","Data":"3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e"}
Oct 30 19:35:55.352024 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:55.351805    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc" event={"ID":"8e7763d2-08f8-4de1-ad2b-b7838b9f881b","Type":"ContainerStarted","Data":"46bbccbe12e69b95932728c18c33ead339b4eceb7454256b06eadef425197abf"}
Oct 30 19:35:55.466770 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:55.466701    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc" podStartSLOduration=1.466683895 podStartE2EDuration="1.466683895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:55.4653802 +0000 UTC m=+3446.353297289" watchObservedRunningTime="2026-10-30 19:35:55.466683895 +0000 UTC m=+3446.354600986"
Oct 30 19:35:55.792626 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:35:55.792535    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:36:02.881860 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:02.881817    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8cp28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:36:02.882550 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:02.882511    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8cp28" podUID="b4b5cee2-4ba7-4d14-bde2-dd266f4706f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:04.791976 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:04.791940    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:36:04.804755 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:04.804730    4267 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:36:05.447883 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:05.447850    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:36:06.046696 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:06.046663    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"]
Oct 30 19:36:06.048870 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:06.048844    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xwx92"]
Oct 30 19:36:07.218766 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:07.218735    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60938420-e80e-45ac-870e-3b81bc3f55e9" path="/var/lib/kubelet/pods/60938420-e80e-45ac-870e-3b81bc3f55e9/volumes"
Oct 30 19:36:17.445145 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.445064    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"]
Oct 30 19:36:17.448469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.448452    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:17.542151 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.542113    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6tb5\" (UniqueName: \"kubernetes.io/projected/adffc756-cb8d-4784-b170-1cc531bd1f68-kube-api-access-f6tb5\") pod \"maas-api-revoke-sub-452b6d7b-8d6d2\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:17.542151 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.542158    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8d6d2\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:17.542437 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.542182    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8d6d2\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:17.553834 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.553799    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:17.553834 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.553806    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:17.554035 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.553986    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:17.554080 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.554055    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:36:17.556382 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.556355    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"]
Oct 30 19:36:17.643068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.643034    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6tb5\" (UniqueName: \"kubernetes.io/projected/adffc756-cb8d-4784-b170-1cc531bd1f68-kube-api-access-f6tb5\") pod \"maas-api-revoke-sub-452b6d7b-8d6d2\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:17.643068 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.643080    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8d6d2\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:17.643303 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.643206    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8d6d2\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:17.643867 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.643844    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8d6d2\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:17.645459 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.645437    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8d6d2\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:17.699480 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.699401    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6tb5\" (UniqueName: \"kubernetes.io/projected/adffc756-cb8d-4784-b170-1cc531bd1f68-kube-api-access-f6tb5\") pod \"maas-api-revoke-sub-452b6d7b-8d6d2\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:17.760963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:17.760922    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:18.143067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:18.143026    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"]
Oct 30 19:36:18.144824 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:36:18.144788    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadffc756_cb8d_4784_b170_1cc531bd1f68.slice/crio-d0a3e91a3fbee65340c0ff500084838b5af942421ed4a609b76a68e6a4bd6d01 WatchSource:0}: Error finding container d0a3e91a3fbee65340c0ff500084838b5af942421ed4a609b76a68e6a4bd6d01: Status 404 returned error can't find the container with id d0a3e91a3fbee65340c0ff500084838b5af942421ed4a609b76a68e6a4bd6d01
Oct 30 19:36:18.504418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:18.504379    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2" event={"ID":"adffc756-cb8d-4784-b170-1cc531bd1f68","Type":"ContainerStarted","Data":"a98a7756304bc0c4ffe8f8293f7d3809712b901801ea6dc2f5d8388ad5381a2a"}
Oct 30 19:36:18.504898 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:18.504441    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2" event={"ID":"adffc756-cb8d-4784-b170-1cc531bd1f68","Type":"ContainerStarted","Data":"d0a3e91a3fbee65340c0ff500084838b5af942421ed4a609b76a68e6a4bd6d01"}
Oct 30 19:36:18.618949 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:18.618901    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2" podStartSLOduration=1.618888326 podStartE2EDuration="1.618888326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:18.618071192 +0000 UTC m=+3469.505988280" watchObservedRunningTime="2026-10-30 19:36:18.618888326 +0000 UTC m=+3469.506805415"
Oct 30 19:36:19.514987 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:19.514945    4267 generic.go:356] "Generic (PLEG): container finished" podID="adffc756-cb8d-4784-b170-1cc531bd1f68" containerID="a98a7756304bc0c4ffe8f8293f7d3809712b901801ea6dc2f5d8388ad5381a2a" exitCode=0
Oct 30 19:36:19.515584 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:19.515003    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2" event={"ID":"adffc756-cb8d-4784-b170-1cc531bd1f68","Type":"ContainerDied","Data":"a98a7756304bc0c4ffe8f8293f7d3809712b901801ea6dc2f5d8388ad5381a2a"}
Oct 30 19:36:20.294515 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:36:20.294003    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:36:21.314981 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.314947    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:21.399460 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.399405    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adffc756-cb8d-4784-b170-1cc531bd1f68-kube-api-access-f6tb5\" (UniqueName: \"kubernetes.io/projected/adffc756-cb8d-4784-b170-1cc531bd1f68-kube-api-access-f6tb5\") pod \"adffc756-cb8d-4784-b170-1cc531bd1f68\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") "
Oct 30 19:36:21.399916 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.399892    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-openshift-ca\") pod \"adffc756-cb8d-4784-b170-1cc531bd1f68\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") "
Oct 30 19:36:21.400142 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.400100    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-xks-ca\") pod \"adffc756-cb8d-4784-b170-1cc531bd1f68\" (UID: \"adffc756-cb8d-4784-b170-1cc531bd1f68\") "
Oct 30 19:36:21.400792 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.400267    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-openshift-ca" pod "adffc756-cb8d-4784-b170-1cc531bd1f68" (UID: "adffc756-cb8d-4784-b170-1cc531bd1f68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:21.401281 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.401259    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:21.406648 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.406605    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-xks-ca" pod "adffc756-cb8d-4784-b170-1cc531bd1f68" (UID: "adffc756-cb8d-4784-b170-1cc531bd1f68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:21.408774 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.408734    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adffc756-cb8d-4784-b170-1cc531bd1f68-kube-api-access-f6tb5" pod "adffc756-cb8d-4784-b170-1cc531bd1f68" (UID: "adffc756-cb8d-4784-b170-1cc531bd1f68"). InnerVolumeSpecName "kube-api-access-f6tb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:21.566521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.562930    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6tb5\" (UniqueName: \"kubernetes.io/projected/adffc756-cb8d-4784-b170-1cc531bd1f68-kube-api-access-f6tb5\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:21.566521 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.562969    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adffc756-cb8d-4784-b170-1cc531bd1f68-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:21.576817 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.576774    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2" event={"ID":"adffc756-cb8d-4784-b170-1cc531bd1f68","Type":"ContainerDied","Data":"d0a3e91a3fbee65340c0ff500084838b5af942421ed4a609b76a68e6a4bd6d01"}
Oct 30 19:36:21.577055 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.577035    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0a3e91a3fbee65340c0ff500084838b5af942421ed4a609b76a68e6a4bd6d01"
Oct 30 19:36:21.577269 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:21.577253    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"
Oct 30 19:36:33.026465 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:33.026423    4267 scope.go:122] "RemoveContainer" containerID="d5b73dd2348275634bdd6517c270aefe73f901d541a04771349310e1ed3af6fe"
Oct 30 19:36:45.592625 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:45.592590    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"]
Oct 30 19:36:49.807969 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.807930    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"]
Oct 30 19:36:49.808373 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.808317    4267 cpu_manager.go:414] "containerMap: removing container" podUID="adffc756-cb8d-4784-b170-1cc531bd1f68" containerName="revoke-keys"
Oct 30 19:36:49.808373 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.808329    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adffc756-cb8d-4784-b170-1cc531bd1f68" containerName="revoke-keys"
Oct 30 19:36:49.808471 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.808413    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="adffc756-cb8d-4784-b170-1cc531bd1f68" containerName="revoke-keys"
Oct 30 19:36:49.812682 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.812663    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:49.865407 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.865372    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:49.899668 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.899626    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sptkf\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:49.899842 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.899811    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsm4x\" (UniqueName: \"kubernetes.io/projected/e44eaabb-8c02-41a8-af28-6ae63edad57e-kube-api-access-zsm4x\") pod \"maas-api-revoke-sub-3e31a506-sptkf\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:49.899960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.899941    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sptkf\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:49.914931 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.914909    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:49.915141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.914934    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:49.915141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.915108    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:36:49.917580 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:49.917555    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"]
Oct 30 19:36:50.000639 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:50.000604    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsm4x\" (UniqueName: \"kubernetes.io/projected/e44eaabb-8c02-41a8-af28-6ae63edad57e-kube-api-access-zsm4x\") pod \"maas-api-revoke-sub-3e31a506-sptkf\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:50.000831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:50.000646    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sptkf\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:50.000831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:50.000687    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sptkf\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:50.001325 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:50.001300    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sptkf\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:50.003117 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:50.003092    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sptkf\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:50.060252 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:50.060176    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsm4x\" (UniqueName: \"kubernetes.io/projected/e44eaabb-8c02-41a8-af28-6ae63edad57e-kube-api-access-zsm4x\") pod \"maas-api-revoke-sub-3e31a506-sptkf\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:50.125970 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:50.125925    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:50.174958 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:36:50.174882    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:36:50.524141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:50.524100    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"]
Oct 30 19:36:50.535522 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:36:50.531189    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44eaabb_8c02_41a8_af28_6ae63edad57e.slice/crio-b94648d5caa06ad53f9f65965747a60f65bed58647aaf76c745f2e2b5ac8f1f4 WatchSource:0}: Error finding container b94648d5caa06ad53f9f65965747a60f65bed58647aaf76c745f2e2b5ac8f1f4: Status 404 returned error can't find the container with id b94648d5caa06ad53f9f65965747a60f65bed58647aaf76c745f2e2b5ac8f1f4
Oct 30 19:36:50.761578 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:50.761534    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf" event={"ID":"e44eaabb-8c02-41a8-af28-6ae63edad57e","Type":"ContainerStarted","Data":"b94648d5caa06ad53f9f65965747a60f65bed58647aaf76c745f2e2b5ac8f1f4"}
Oct 30 19:36:51.789166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:51.788751    4267 generic.go:356] "Generic (PLEG): container finished" podID="e44eaabb-8c02-41a8-af28-6ae63edad57e" containerID="25d9f338a61e05f7895f73a78fe4d54d3b3ee5a40824ee22c8632c55391d1a25" exitCode=0
Oct 30 19:36:51.789166 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:51.788805    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf" event={"ID":"e44eaabb-8c02-41a8-af28-6ae63edad57e","Type":"ContainerDied","Data":"25d9f338a61e05f7895f73a78fe4d54d3b3ee5a40824ee22c8632c55391d1a25"}
Oct 30 19:36:53.032435 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.032404    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:53.068796 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.068737    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-xks-ca\") pod \"e44eaabb-8c02-41a8-af28-6ae63edad57e\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") "
Oct 30 19:36:53.068940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.068817    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-openshift-ca\") pod \"e44eaabb-8c02-41a8-af28-6ae63edad57e\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") "
Oct 30 19:36:53.068940 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.068879    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44eaabb-8c02-41a8-af28-6ae63edad57e-kube-api-access-zsm4x\" (UniqueName: \"kubernetes.io/projected/e44eaabb-8c02-41a8-af28-6ae63edad57e-kube-api-access-zsm4x\") pod \"e44eaabb-8c02-41a8-af28-6ae63edad57e\" (UID: \"e44eaabb-8c02-41a8-af28-6ae63edad57e\") "
Oct 30 19:36:53.069219 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.069194    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-openshift-ca" pod "e44eaabb-8c02-41a8-af28-6ae63edad57e" (UID: "e44eaabb-8c02-41a8-af28-6ae63edad57e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:53.070741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.070720    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-xks-ca" pod "e44eaabb-8c02-41a8-af28-6ae63edad57e" (UID: "e44eaabb-8c02-41a8-af28-6ae63edad57e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:53.071026 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.071004    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44eaabb-8c02-41a8-af28-6ae63edad57e-kube-api-access-zsm4x" pod "e44eaabb-8c02-41a8-af28-6ae63edad57e" (UID: "e44eaabb-8c02-41a8-af28-6ae63edad57e"). InnerVolumeSpecName "kube-api-access-zsm4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:53.170141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.170100    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:53.170141 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.170137    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsm4x\" (UniqueName: \"kubernetes.io/projected/e44eaabb-8c02-41a8-af28-6ae63edad57e-kube-api-access-zsm4x\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:53.170340 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.170153    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44eaabb-8c02-41a8-af28-6ae63edad57e-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:53.830454 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.830417    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf" event={"ID":"e44eaabb-8c02-41a8-af28-6ae63edad57e","Type":"ContainerDied","Data":"b94648d5caa06ad53f9f65965747a60f65bed58647aaf76c745f2e2b5ac8f1f4"}
Oct 30 19:36:53.830454 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.830440    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sptkf"
Oct 30 19:36:53.830961 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:36:53.830468    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b94648d5caa06ad53f9f65965747a60f65bed58647aaf76c745f2e2b5ac8f1f4"
Oct 30 19:37:00.613297 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.613222    4267 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc" podUID="8e7763d2-08f8-4de1-ad2b-b7838b9f881b" containerName="main" containerID="cri-o://3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e" gracePeriod=45
Oct 30 19:37:00.868850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.868759    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:37:00.869919 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.869897    4267 generic.go:356] "Generic (PLEG): container finished" podID="8e7763d2-08f8-4de1-ad2b-b7838b9f881b" containerID="3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e" exitCode=0
Oct 30 19:37:00.870021 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.869966    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc" event={"ID":"8e7763d2-08f8-4de1-ad2b-b7838b9f881b","Type":"ContainerDied","Data":"3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e"}
Oct 30 19:37:00.870021 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.870012    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc" event={"ID":"8e7763d2-08f8-4de1-ad2b-b7838b9f881b","Type":"ContainerDied","Data":"46bbccbe12e69b95932728c18c33ead339b4eceb7454256b06eadef425197abf"}
Oct 30 19:37:00.870092 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.870030    4267 scope.go:122] "RemoveContainer" containerID="3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e"
Oct 30 19:37:00.878827 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.878806    4267 scope.go:122] "RemoveContainer" containerID="3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e"
Oct 30 19:37:00.879126 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:37:00.879102    4267 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e\": container with ID starting with 3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e not found: ID does not exist" containerID="3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e"
Oct 30 19:37:00.879180 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.879134    4267 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e"} err="failed to get container status \"3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e\": rpc error: code = NotFound desc = could not find container \"3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e\": container with ID starting with 3d542a45dcd12a11ee8f1034bf4324e91303cd15a5dde9c420000be6cffaf83e not found: ID does not exist"
Oct 30 19:37:00.950054 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950013    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-home\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-home\") pod \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") "
Oct 30 19:37:00.950266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950080    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-dshm\") pod \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") "
Oct 30 19:37:00.950266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950122    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-kube-api-access-dbg8q\" (UniqueName: \"kubernetes.io/projected/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-kube-api-access-dbg8q\") pod \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") "
Oct 30 19:37:00.950266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950143    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-model-cache\") pod \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") "
Oct 30 19:37:00.950266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950183    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tmp-dir\") pod \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") "
Oct 30 19:37:00.950266 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950259    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tls-certs\") pod \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\" (UID: \"8e7763d2-08f8-4de1-ad2b-b7838b9f881b\") "
Oct 30 19:37:00.950599 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950379    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-home" pod "8e7763d2-08f8-4de1-ad2b-b7838b9f881b" (UID: "8e7763d2-08f8-4de1-ad2b-b7838b9f881b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:00.950599 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950528    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tmp-dir" pod "8e7763d2-08f8-4de1-ad2b-b7838b9f881b" (UID: "8e7763d2-08f8-4de1-ad2b-b7838b9f881b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:00.950599 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950537    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-model-cache" pod "8e7763d2-08f8-4de1-ad2b-b7838b9f881b" (UID: "8e7763d2-08f8-4de1-ad2b-b7838b9f881b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:00.950741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950645    4267 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-home\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:00.950741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950659    4267 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-model-cache\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:00.950741 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.950695    4267 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tmp-dir\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:00.952444 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.952412    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-dshm" pod "8e7763d2-08f8-4de1-ad2b-b7838b9f881b" (UID: "8e7763d2-08f8-4de1-ad2b-b7838b9f881b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:00.952601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.952450    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-kube-api-access-dbg8q" pod "8e7763d2-08f8-4de1-ad2b-b7838b9f881b" (UID: "8e7763d2-08f8-4de1-ad2b-b7838b9f881b"). InnerVolumeSpecName "kube-api-access-dbg8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:00.952601 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:00.952533    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tls-certs" pod "8e7763d2-08f8-4de1-ad2b-b7838b9f881b" (UID: "8e7763d2-08f8-4de1-ad2b-b7838b9f881b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:01.051959 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:01.051912    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbg8q\" (UniqueName: \"kubernetes.io/projected/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-kube-api-access-dbg8q\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:01.051959 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:01.051956    4267 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-tls-certs\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:01.052193 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:01.051971    4267 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e7763d2-08f8-4de1-ad2b-b7838b9f881b-dshm\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:01.878272 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:01.878239    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"
Oct 30 19:37:02.040022 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:02.039982    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"]
Oct 30 19:37:02.089573 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:02.089535    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-00230e87-kserve-75cf4b674f-rtftc"]
Oct 30 19:37:03.258398 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:03.258340    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7763d2-08f8-4de1-ad2b-b7838b9f881b" path="/var/lib/kubelet/pods/8e7763d2-08f8-4de1-ad2b-b7838b9f881b/volumes"
Oct 30 19:37:19.416406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.416369    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"]
Oct 30 19:37:19.416965 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.416900    4267 cpu_manager.go:414] "containerMap: removing container" podUID="8e7763d2-08f8-4de1-ad2b-b7838b9f881b" containerName="main"
Oct 30 19:37:19.416965 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.416921    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7763d2-08f8-4de1-ad2b-b7838b9f881b" containerName="main"
Oct 30 19:37:19.417086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.416976    4267 cpu_manager.go:414] "containerMap: removing container" podUID="e44eaabb-8c02-41a8-af28-6ae63edad57e" containerName="revoke-keys"
Oct 30 19:37:19.417086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.416985    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44eaabb-8c02-41a8-af28-6ae63edad57e" containerName="revoke-keys"
Oct 30 19:37:19.417086 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.417074    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44eaabb-8c02-41a8-af28-6ae63edad57e" containerName="revoke-keys"
Oct 30 19:37:19.417247 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.417089    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7763d2-08f8-4de1-ad2b-b7838b9f881b" containerName="main"
Oct 30 19:37:19.420351 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.420329    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:19.475350 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.475311    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"]
Oct 30 19:37:19.505243 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.505209    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4fp9\" (UniqueName: \"kubernetes.io/projected/e24dcc6a-1c89-4709-b79e-2cc34286bd67-kube-api-access-b4fp9\") pod \"maas-api-revoke-sub-2bacaa8f-z98nf\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:19.505443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.505262    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bacaa8f-z98nf\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:19.505443 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.505360    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bacaa8f-z98nf\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:19.521848 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.521817    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:19.521954 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.521872    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nqq9m\""
Oct 30 19:37:19.521954 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.521873    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:19.522358 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.522341    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:19.605999 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.605959    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bacaa8f-z98nf\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:19.606198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.606025    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bacaa8f-z98nf\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:19.606198 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.606088    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fp9\" (UniqueName: \"kubernetes.io/projected/e24dcc6a-1c89-4709-b79e-2cc34286bd67-kube-api-access-b4fp9\") pod \"maas-api-revoke-sub-2bacaa8f-z98nf\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:19.606831 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.606800    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bacaa8f-z98nf\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:19.608469 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.608445    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bacaa8f-z98nf\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:19.663096 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.663064    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fp9\" (UniqueName: \"kubernetes.io/projected/e24dcc6a-1c89-4709-b79e-2cc34286bd67-kube-api-access-b4fp9\") pod \"maas-api-revoke-sub-2bacaa8f-z98nf\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:19.734401 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:19.734359    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:20.106018 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:20.105988    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"]
Oct 30 19:37:20.107403 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:37:20.107373    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24dcc6a_1c89_4709_b79e_2cc34286bd67.slice/crio-4fffaec87817952a28499039ecf98c46dc8d21d8fe0f77dc7bc6f546139736b0 WatchSource:0}: Error finding container 4fffaec87817952a28499039ecf98c46dc8d21d8fe0f77dc7bc6f546139736b0: Status 404 returned error can't find the container with id 4fffaec87817952a28499039ecf98c46dc8d21d8fe0f77dc7bc6f546139736b0
Oct 30 19:37:20.158027 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:37:20.157992    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:37:21.024818 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:21.024788    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf" event={"ID":"e24dcc6a-1c89-4709-b79e-2cc34286bd67","Type":"ContainerStarted","Data":"cf1a3f57cce3058a60ac6d384a7bc753b7389b81cd388334a98e293f7669c5c2"}
Oct 30 19:37:21.025172 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:21.024849    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf" event={"ID":"e24dcc6a-1c89-4709-b79e-2cc34286bd67","Type":"ContainerStarted","Data":"4fffaec87817952a28499039ecf98c46dc8d21d8fe0f77dc7bc6f546139736b0"}
Oct 30 19:37:22.032331 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.032290    4267 generic.go:356] "Generic (PLEG): container finished" podID="e24dcc6a-1c89-4709-b79e-2cc34286bd67" containerID="cf1a3f57cce3058a60ac6d384a7bc753b7389b81cd388334a98e293f7669c5c2" exitCode=0
Oct 30 19:37:22.032816 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.032375    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf" event={"ID":"e24dcc6a-1c89-4709-b79e-2cc34286bd67","Type":"ContainerDied","Data":"cf1a3f57cce3058a60ac6d384a7bc753b7389b81cd388334a98e293f7669c5c2"}
Oct 30 19:37:22.172416 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.172386    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:22.333417 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.333336    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e24dcc6a-1c89-4709-b79e-2cc34286bd67-kube-api-access-b4fp9\" (UniqueName: \"kubernetes.io/projected/e24dcc6a-1c89-4709-b79e-2cc34286bd67-kube-api-access-b4fp9\") pod \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") "
Oct 30 19:37:22.333616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.333447    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-openshift-ca\") pod \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") "
Oct 30 19:37:22.333616 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.333520    4267 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-xks-ca\") pod \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\" (UID: \"e24dcc6a-1c89-4709-b79e-2cc34286bd67\") "
Oct 30 19:37:22.333824 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.333798    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-openshift-ca" pod "e24dcc6a-1c89-4709-b79e-2cc34286bd67" (UID: "e24dcc6a-1c89-4709-b79e-2cc34286bd67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:22.335645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.335617    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e24dcc6a-1c89-4709-b79e-2cc34286bd67-kube-api-access-b4fp9" pod "e24dcc6a-1c89-4709-b79e-2cc34286bd67" (UID: "e24dcc6a-1c89-4709-b79e-2cc34286bd67"). InnerVolumeSpecName "kube-api-access-b4fp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:22.335645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.335629    4267 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-xks-ca" pod "e24dcc6a-1c89-4709-b79e-2cc34286bd67" (UID: "e24dcc6a-1c89-4709-b79e-2cc34286bd67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:22.435082 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.435040    4267 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4fp9\" (UniqueName: \"kubernetes.io/projected/e24dcc6a-1c89-4709-b79e-2cc34286bd67-kube-api-access-b4fp9\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:22.435082 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.435076    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-openshift-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:22.435082 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:22.435086    4267 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e24dcc6a-1c89-4709-b79e-2cc34286bd67-maas-api-xks-ca\") on node \"ip-10-0-131-68.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:37:23.039138 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:23.039101    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf" event={"ID":"e24dcc6a-1c89-4709-b79e-2cc34286bd67","Type":"ContainerDied","Data":"4fffaec87817952a28499039ecf98c46dc8d21d8fe0f77dc7bc6f546139736b0"}
Oct 30 19:37:23.039674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:23.039162    4267 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fffaec87817952a28499039ecf98c46dc8d21d8fe0f77dc7bc6f546139736b0"
Oct 30 19:37:23.039674 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:23.039111    4267 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bacaa8f-z98nf"
Oct 30 19:37:33.047027 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:33.046988    4267 scope.go:122] "RemoveContainer" containerID="26a798a61df6da662cb2a76476927d38c8b1ee8a7d3f9df50c68276923aa0682"
Oct 30 19:37:33.057549 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:33.057525    4267 scope.go:122] "RemoveContainer" containerID="d3e41e5cfb31d6197892bfc8fefc40a90ab3ca94be2beb2fc28d13a0cc9c0d9b"
Oct 30 19:37:41.614577 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.614473    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"]
Oct 30 19:37:41.614964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.614844    4267 cpu_manager.go:414] "containerMap: removing container" podUID="e24dcc6a-1c89-4709-b79e-2cc34286bd67" containerName="revoke-keys"
Oct 30 19:37:41.614964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.614856    4267 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e24dcc6a-1c89-4709-b79e-2cc34286bd67" containerName="revoke-keys"
Oct 30 19:37:41.614964 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.614923    4267 memory_manager.go:365] "RemoveStaleState removing state" podUID="e24dcc6a-1c89-4709-b79e-2cc34286bd67" containerName="revoke-keys"
Oct 30 19:37:41.617860 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.617840    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.699430 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.699393    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebaf99a0-9a1d-4c9d-9988-4b4097637835-maas-api-tls\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.699657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.699470    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebaf99a0-9a1d-4c9d-9988-4b4097637835-openshift-service-ca\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.699657 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.699598    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnkvj\" (UniqueName: \"kubernetes.io/projected/ebaf99a0-9a1d-4c9d-9988-4b4097637835-kube-api-access-cnkvj\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.699788 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.699674    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebaf99a0-9a1d-4c9d-9988-4b4097637835-metrics-certs\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.730838 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.730794    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:41.731050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.730835    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 19:37:41.731050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.730859    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9wflp\""
Oct 30 19:37:41.731050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.730836    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:41.731050 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.730936    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:37:41.733548 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.733520    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"]
Oct 30 19:37:41.800371 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.800334    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebaf99a0-9a1d-4c9d-9988-4b4097637835-maas-api-tls\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.800596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.800385    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebaf99a0-9a1d-4c9d-9988-4b4097637835-openshift-service-ca\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.800596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.800416    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnkvj\" (UniqueName: \"kubernetes.io/projected/ebaf99a0-9a1d-4c9d-9988-4b4097637835-kube-api-access-cnkvj\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.800596 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.800445    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebaf99a0-9a1d-4c9d-9988-4b4097637835-metrics-certs\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.801131 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.801102    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebaf99a0-9a1d-4c9d-9988-4b4097637835-openshift-service-ca\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.802992 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.802973    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebaf99a0-9a1d-4c9d-9988-4b4097637835-metrics-certs\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.803057 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.803011    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebaf99a0-9a1d-4c9d-9988-4b4097637835-maas-api-tls\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.858169 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.858132    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnkvj\" (UniqueName: \"kubernetes.io/projected/ebaf99a0-9a1d-4c9d-9988-4b4097637835-kube-api-access-cnkvj\") pod \"maas-api-595cc648c-xqnl8\" (UID: \"ebaf99a0-9a1d-4c9d-9988-4b4097637835\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:41.930846 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:41.930746    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:42.320316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:42.320287    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"]
Oct 30 19:37:42.322203 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:37:42.322165    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebaf99a0_9a1d_4c9d_9988_4b4097637835.slice/crio-87807b0ef6db5ceedb63b68f6b849edf4152cfc0bf5877ccc796eb994cc6de1a WatchSource:0}: Error finding container 87807b0ef6db5ceedb63b68f6b849edf4152cfc0bf5877ccc796eb994cc6de1a: Status 404 returned error can't find the container with id 87807b0ef6db5ceedb63b68f6b849edf4152cfc0bf5877ccc796eb994cc6de1a
Oct 30 19:37:42.323645 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:42.323626    4267 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:37:43.142895 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:43.142858    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8" event={"ID":"ebaf99a0-9a1d-4c9d-9988-4b4097637835","Type":"ContainerStarted","Data":"87807b0ef6db5ceedb63b68f6b849edf4152cfc0bf5877ccc796eb994cc6de1a"}
Oct 30 19:37:44.152402 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:44.152357    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8" event={"ID":"ebaf99a0-9a1d-4c9d-9988-4b4097637835","Type":"ContainerStarted","Data":"26b1cb8384ed52a698c1d46b469871e37ae4cd5f740d62651b05c062579cc22d"}
Oct 30 19:37:44.152986 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:44.152527    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:44.269864 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:44.269792    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8" podStartSLOduration=2.385754268 podStartE2EDuration="3.269772995s" totalImagesPullingTime="884.018727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:44.26759324 +0000 UTC m=+3555.155510330" watchObservedRunningTime="2026-10-30 19:37:44.269772995 +0000 UTC m=+3555.157690085"
Oct 30 19:37:50.167006 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:37:50.166961    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:37:50.178511 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:50.178471    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-xqnl8"
Oct 30 19:37:56.053518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:56.052703    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"]
Oct 30 19:37:56.111518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:56.111251    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p4zv7"]
Oct 30 19:37:56.601519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:56.601337    4267 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-x8kbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:56.601519 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:56.601442    4267 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-x8kbj" podUID="38d93a40-3273-466a-9475-fa0960ed3e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:57.250518 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:37:57.244777    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e91d125-96d0-4a51-acb8-bd6060ff2766" path="/var/lib/kubelet/pods/8e91d125-96d0-4a51-acb8-bd6060ff2766/volumes"
Oct 30 19:38:17.047854 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:38:17.047816    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"]
Oct 30 19:38:17.097472 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:38:17.097431    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6snnd"]
Oct 30 19:38:17.220138 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:38:17.220107    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102f6129-2699-4328-8e3d-f1f11ebc5f99" path="/var/lib/kubelet/pods/102f6129-2699-4328-8e3d-f1f11ebc5f99/volumes"
Oct 30 19:38:20.157444 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:38:20.157401    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:38:47.284067 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:38:47.284034    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84f7b9c6bb-t78t4_e593c4e1-00ce-4f98-acbd-a1a0beb50b07/authorino/0.log"
Oct 30 19:38:50.157415 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:38:50.157375    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:39:01.211804 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:01.211776    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb69c88fb-dqpcm_276385b5-61ba-4c52-a013-ba32fdccafdc/copy-manifests/0.log"
Oct 30 19:39:01.277893 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:01.277865    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb69c88fb-dqpcm_276385b5-61ba-4c52-a013-ba32fdccafdc/manager/0.log"
Oct 30 19:39:02.594960 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:02.594925    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-xqnl8_ebaf99a0-9a1d-4c9d-9988-4b4097637835/maas-api/0.log"
Oct 30 19:39:03.117451 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:03.117423    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2bacaa8f-z98nf_e24dcc6a-1c89-4709-b79e-2cc34286bd67/revoke-keys/0.log"
Oct 30 19:39:03.346699 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:03.346667    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-sptkf_e44eaabb-8c02-41a8-af28-6ae63edad57e/revoke-keys/0.log"
Oct 30 19:39:03.567930 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:03.567897    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-8d6d2_adffc756-cb8d-4784-b170-1cc531bd1f68/revoke-keys/0.log"
Oct 30 19:39:04.015963 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:04.015931    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9f0d1ad8-xrcbm_5bc70997-f350-45bc-bdb7-8f32e8c75b11/revoke-keys/0.log"
Oct 30 19:39:06.012392 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:06.012353    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84f7b9c6bb-t78t4_e593c4e1-00ce-4f98-acbd-a1a0beb50b07/authorino/0.log"
Oct 30 19:39:07.031174 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:07.031144    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9n45v_22c77e4c-cf9d-470b-8ab2-c1cad5221b8c/kuadrant-console-plugin/0.log"
Oct 30 19:39:07.282308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:07.282224    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-8wbf4_28f82707-5133-4bd8-b36a-62042290ac08/manager/0.log"
Oct 30 19:39:09.554374 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:09.554345    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fb96896c4-ph4kk_5599feb7-82b3-4fd6-a592-f7b671791e6d/kube-auth-proxy/0.log"
Oct 30 19:39:10.238316 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:10.238286    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-864kk_96d47ea1-b235-45d8-8573-0dae95cd754d/istio-proxy/0.log"
Oct 30 19:39:10.562952 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:10.562866    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-8cp28_b4b5cee2-4ba7-4d14-bde2-dd266f4706f6/istio-proxy/0.log"
Oct 30 19:39:11.278765 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:11.278679    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-m6v84_1650ffd0-6942-4650-8795-2de75a4fbc87/istio-proxy/0.log"
Oct 30 19:39:12.014046 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:12.014016    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-x8kbj_38d93a40-3273-466a-9475-fa0960ed3e35/istio-proxy/0.log"
Oct 30 19:39:13.623850 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:13.623816    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg_46d1bbb4-c192-4a1e-a791-cd06f0976714/main/0.log"
Oct 30 19:39:14.590217 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:14.590186    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp_26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553/main/0.log"
Oct 30 19:39:15.094365 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:15.094339    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb_e108a863-9bb1-46e7-a1ca-93c23dc91d92/main/0.log"
Oct 30 19:39:16.136346 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:16.136311    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84f7b9c6bb-t78t4_e593c4e1-00ce-4f98-acbd-a1a0beb50b07/authorino/0.log"
Oct 30 19:39:20.157162 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:39:20.157126    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:39:28.408087 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:28.408057    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb69c88fb-dqpcm_276385b5-61ba-4c52-a013-ba32fdccafdc/copy-manifests/0.log"
Oct 30 19:39:28.476993 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:28.476963    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cb69c88fb-dqpcm_276385b5-61ba-4c52-a013-ba32fdccafdc/manager/0.log"
Oct 30 19:39:29.563167 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:29.563131    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-xqnl8_ebaf99a0-9a1d-4c9d-9988-4b4097637835/maas-api/0.log"
Oct 30 19:39:30.102130 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:30.102095    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2bacaa8f-z98nf_e24dcc6a-1c89-4709-b79e-2cc34286bd67/revoke-keys/0.log"
Oct 30 19:39:30.321534 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:30.321482    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-sptkf_e44eaabb-8c02-41a8-af28-6ae63edad57e/revoke-keys/0.log"
Oct 30 19:39:30.539278 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:30.539242    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-8d6d2_adffc756-cb8d-4784-b170-1cc531bd1f68/revoke-keys/0.log"
Oct 30 19:39:30.989308 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:30.989278    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9f0d1ad8-xrcbm_5bc70997-f350-45bc-bdb7-8f32e8c75b11/revoke-keys/0.log"
Oct 30 19:39:32.679915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:32.679880    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84f7b9c6bb-t78t4_e593c4e1-00ce-4f98-acbd-a1a0beb50b07/authorino/0.log"
Oct 30 19:39:33.049482 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:33.049445    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"]
Oct 30 19:39:33.094395 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:33.094359    4267 scope.go:122] "RemoveContainer" containerID="779c49d61557d368004a2003aeab1614fbb9219104da11628dd56f151b24ada2"
Oct 30 19:39:33.098379 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:33.098354    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f0d1ad8-xrcbm"]
Oct 30 19:39:33.103617 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:33.103598    4267 scope.go:122] "RemoveContainer" containerID="5177b3f29e818b0e1e4db1b3b8026f45ce21ce7f21d15b0c81cd01e690ff6e25"
Oct 30 19:39:33.219051 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:33.219020    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc70997-f350-45bc-bdb7-8f32e8c75b11" path="/var/lib/kubelet/pods/5bc70997-f350-45bc-bdb7-8f32e8c75b11/volumes"
Oct 30 19:39:33.683903 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:33.683877    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9n45v_22c77e4c-cf9d-470b-8ab2-c1cad5221b8c/kuadrant-console-plugin/0.log"
Oct 30 19:39:33.919771 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:33.919738    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-8wbf4_28f82707-5133-4bd8-b36a-62042290ac08/manager/0.log"
Oct 30 19:39:35.915171 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:35.915138    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fb96896c4-ph4kk_5599feb7-82b3-4fd6-a592-f7b671791e6d/kube-auth-proxy/0.log"
Oct 30 19:39:36.588253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:36.588221    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-864kk_96d47ea1-b235-45d8-8573-0dae95cd754d/istio-proxy/0.log"
Oct 30 19:39:36.869418 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:36.869343    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-8cp28_b4b5cee2-4ba7-4d14-bde2-dd266f4706f6/istio-proxy/0.log"
Oct 30 19:39:37.587452 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:37.587414    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-m6v84_1650ffd0-6942-4650-8795-2de75a4fbc87/istio-proxy/0.log"
Oct 30 19:39:38.309393 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:38.309364    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-x8kbj_38d93a40-3273-466a-9475-fa0960ed3e35/istio-proxy/0.log"
Oct 30 19:39:39.638309 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:39.638280    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9mptg_46d1bbb4-c192-4a1e-a791-cd06f0976714/main/0.log"
Oct 30 19:39:40.567140 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:40.567108    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8s9wnp_26e4fdf8-054d-4f5f-bb3a-9bdb3cc3d553/main/0.log"
Oct 30 19:39:41.067812 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:39:41.067764    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-wr8lb_e108a863-9bb1-46e7-a1ca-93c23dc91d92/main/0.log"
Oct 30 19:39:50.161571 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:39:50.161530    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:40:20.156840 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:40:20.156798    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:40:33.127481 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:40:33.127450    4267 scope.go:122] "RemoveContainer" containerID="6012efe150c06006fd0cd3d2cf5692a31e078fa78140c4e77d8744ac86e026be"
Oct 30 19:40:50.158138 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:40:50.158047    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:41:03.129232 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:03.129198    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lwdl6_6bbdacee-0942-49d7-b3eb-9730db6a9a73/global-pull-secret-syncer/0.log"
Oct 30 19:41:05.409915 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:05.409871    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sn8g8_deae8a19-42d0-4d66-8029-f2f131584fbc/konnectivity-agent/0.log"
Oct 30 19:41:06.357748 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:06.357710    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-68.us-west-1.compute.internal_ee243aa474c3eaea9adbefcd420da07b/haproxy/0.log"
Oct 30 19:41:11.266564 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:11.266530    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-84f7b9c6bb-t78t4_e593c4e1-00ce-4f98-acbd-a1a0beb50b07/authorino/0.log"
Oct 30 19:41:11.981753 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:11.981716    4267 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"]
Oct 30 19:41:11.985202 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:11.985177    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.037565 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.037512    4267 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vh98j\"/\"default-dockercfg-wmjvf\""
Oct 30 19:41:12.087083 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.087050    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vh98j\"/\"openshift-service-ca.crt\""
Oct 30 19:41:12.087253 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.087145    4267 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vh98j\"/\"kube-root-ca.crt\""
Oct 30 19:41:12.089751 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.089730    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"]
Oct 30 19:41:12.091570 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.091549    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plq6j\" (UniqueName: \"kubernetes.io/projected/3837c585-5507-43b6-872f-e9d0c2ea7052-kube-api-access-plq6j\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.091685 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.091607    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-proc\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.091735 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.091716    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-lib-modules\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.091773 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.091740    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-sys\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.091869 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.091854    4267 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-podres\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.193183 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.193141    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-podres\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.193406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.193206    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plq6j\" (UniqueName: \"kubernetes.io/projected/3837c585-5507-43b6-872f-e9d0c2ea7052-kube-api-access-plq6j\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.193406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.193260    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-proc\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.193406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.193301    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-podres\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.193406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.193309    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-lib-modules\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.193406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.193380    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-lib-modules\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.193406 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.193385    4267 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-sys\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.193687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.193440    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-sys\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.193687 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.193376    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3837c585-5507-43b6-872f-e9d0c2ea7052-proc\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.251204 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.251121    4267 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plq6j\" (UniqueName: \"kubernetes.io/projected/3837c585-5507-43b6-872f-e9d0c2ea7052-kube-api-access-plq6j\") pod \"perf-node-gather-daemonset-m9pgj\" (UID: \"3837c585-5507-43b6-872f-e9d0c2ea7052\") " pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.298105 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.298068    4267 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:12.427858 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.427830    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9n45v_22c77e4c-cf9d-470b-8ab2-c1cad5221b8c/kuadrant-console-plugin/0.log"
Oct 30 19:41:12.676192 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.676165    4267 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"]
Oct 30 19:41:12.678064 ip-10-0-131-68 kubenswrapper[4267]: W1030 19:41:12.678035    4267 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3837c585_5507_43b6_872f_e9d0c2ea7052.slice/crio-5e4962633cb5446be48c0aa90db86a754207dd628be28a72f619b6e418405de9 WatchSource:0}: Error finding container 5e4962633cb5446be48c0aa90db86a754207dd628be28a72f619b6e418405de9: Status 404 returned error can't find the container with id 5e4962633cb5446be48c0aa90db86a754207dd628be28a72f619b6e418405de9
Oct 30 19:41:12.862368 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:12.862331    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-8wbf4_28f82707-5133-4bd8-b36a-62042290ac08/manager/0.log"
Oct 30 19:41:13.349225 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:13.349187    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj" event={"ID":"3837c585-5507-43b6-872f-e9d0c2ea7052","Type":"ContainerStarted","Data":"6d97bde9e62e1509bd90216b9370b650eb932309a8219cad7862bbeba036dce0"}
Oct 30 19:41:13.349697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:13.349249    4267 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj" event={"ID":"3837c585-5507-43b6-872f-e9d0c2ea7052","Type":"ContainerStarted","Data":"5e4962633cb5446be48c0aa90db86a754207dd628be28a72f619b6e418405de9"}
Oct 30 19:41:13.349697 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:13.349368    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:13.472878 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:13.472817    4267 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj" podStartSLOduration=2.472802622 podStartE2EDuration="2.472802622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:13.470662622 +0000 UTC m=+3764.358579711" watchObservedRunningTime="2026-10-30 19:41:13.472802622 +0000 UTC m=+3764.360719710"
Oct 30 19:41:19.365023 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:19.364993    4267 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vh98j/perf-node-gather-daemonset-m9pgj"
Oct 30 19:41:20.160890 ip-10-0-131-68 kubenswrapper[4267]: E1030 19:41:20.160847    4267 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2km4w" containerName="extractor"
Oct 30 19:41:21.061695 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:21.061661    4267 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"]
Oct 30 19:41:21.111202 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:21.111167    4267 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8d6d2"]
Oct 30 19:41:21.219391 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:21.219357    4267 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adffc756-cb8d-4784-b170-1cc531bd1f68" path="/var/lib/kubelet/pods/adffc756-cb8d-4784-b170-1cc531bd1f68/volumes"
Oct 30 19:41:24.373341 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:24.373312    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s2xhg_761f0730-7b45-4b34-92c9-460d84a7f2be/node-exporter/0.log"
Oct 30 19:41:24.790449 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:24.790407    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s2xhg_761f0730-7b45-4b34-92c9-460d84a7f2be/kube-rbac-proxy/0.log"
Oct 30 19:41:25.210632 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:25.210600    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s2xhg_761f0730-7b45-4b34-92c9-460d84a7f2be/init-textfile/0.log"
Oct 30 19:41:29.871758 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:29.871716    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-s9xdm_9db20512-6277-4144-8065-2f7af76decd2/prometheus-operator/0.log"
Oct 30 19:41:30.285074 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:30.285038    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-s9xdm_9db20512-6277-4144-8065-2f7af76decd2/kube-rbac-proxy/0.log"
Oct 30 19:41:30.711457 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:30.711398    4267 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-p4g59_904bb0c9-c485-4f03-81f1-a0c885b2b8bc/prometheus-operator-admission-webhook/0.log"
Oct 30 19:41:33.147446 ip-10-0-131-68 kubenswrapper[4267]: I1030 19:41:33.147403    4267 scope.go:122] "RemoveContainer" containerID="0613b274496466f069d76848727a6a7a689f8889dfeac80907580fa107421d8e"